> ## Documentation Index
> Fetch the complete documentation index at: https://www.bolna.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Use Baseten Model APIs or dedicated deployments in Bolna

> Use Baseten Model APIs or your own Truss deployment on dedicated GPUs as the LLM for a Bolna voice agent. Endpoints, model slugs, Fast variants and API keys.

[Baseten](https://www.baseten.co) is a production inference platform with two ways in, and they need different values in Bolna:

* **Model APIs**: a supported set of popular open models that Baseten runs for you, called by model slug at one shared URL. Nothing to deploy.
* **Dedicated deployments**: you deploy any model, including fine-tunes, to GPUs reserved for you with a short `config.yaml` and the Truss CLI. Each deployment gets its own URL.

Choose Model APIs to get running in minutes. Choose a dedicated deployment when you need a model Model APIs doesn't offer, your own weights, or capacity nobody else shares.

***

## Get a Baseten API key

Create a key on the [API keys](https://app.baseten.co/settings/api_keys) page of your Baseten workspace settings. Bolna sends it as `Authorization: Bearer <key>`, which is how Baseten's OpenAI-compatible endpoints authenticate.

***

## Option A: Model APIs

**1. Pick a model slug**

Open Baseten's [Model APIs](https://docs.baseten.co/development/model-apis/overview) list and copy the slug, for example `zai-org/GLM-5.3`. Some models also have a **Fast** variant: the same weights with dedicated capacity for higher sustained throughput, under a separate slug with its own pricing and rate limits.

**2. Choose Baseten in Bolna**

Open **Settings → Integrations → LLM → Custom LLM → Add model**, or choose **+ Add custom LLM** from the **LLM Provider** dropdown on an agent. Select **Baseten**.

![Bolna Add custom model dialog with Baseten selected and inference.baseten.co/v1 shown as the endpoint placeholder](https://mintcdn.com/bolna-54a2d4fe/0EXypen329Eb5XE2/images/custom-llm/providers/baseten-selected.png?fit=max\&auto=format\&n=0EXypen329Eb5XE2\&q=85\&s=318bf2ac02cf9a045f32ddd2f4fe5385)

Baseten selected: Model APIs endpoint as placeholder, hint for dedicated deployments

**3. Register the model**

**Dashboard**

* **Model name**: the slug, for example `zai-org/GLM-5.3`
* **Endpoint URL**: `https://inference.baseten.co/v1`
* **API key**: your Baseten API key
* **Family**: for example `glm`

![Add custom model dialog with Baseten selected, model zai-org/GLM-5.3, endpoint inference.baseten.co/v1, masked key and family glm](https://mintcdn.com/bolna-54a2d4fe/aV72vktown5Rn3f1/images/custom-llm/providers/baseten-filled.png?fit=max\&auto=format\&n=aV72vktown5Rn3f1\&q=85\&s=c4a9f3983487cd096ec8123ab4be2195)

Baseten Model APIs, filled in

**API**

```bash theme={"system"}
curl --request POST \
  --url https://api.bolna.ai/user/model/custom \
  --header 'Authorization: Bearer <BOLNA_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "custom_model_name": "zai-org/GLM-5.3",
    "custom_model_url": "https://inference.baseten.co/v1",
    "custom_model_key": "<BASETEN_API_KEY>",
    "custom_model_family": "glm"
  }'
```

> [!NOTE]
>
> Baseten's context and output limits for a Model API can be lower than the model's advertised maximum. Check the limits in the Model APIs table before writing a very long system prompt.

***

## Option B: A dedicated deployment

**1. Install Truss and sign in**

```bash theme={"system"}
pip install --upgrade truss
truss login --browser
```

**2. Write a config.yaml**

Follow Baseten's [Deploy your first model](https://docs.baseten.co/examples/deploy-your-first-model) guide. The config names the Hugging Face checkpoint (`checkpoint_repository`), the GPU and the inference engine. With the TensorRT-LLM engine, no Python code is needed.

**3. Deploy**

```bash theme={"system"}
truss push
```

Truss prints a model ID and a deployment ID. Wait until the deployment shows **Active** in the Baseten dashboard.

**4. Register the deployment in Bolna**

Choose **Baseten** in the Add custom model dialog and replace the placeholder with your deployment's OpenAI-compatible URL:

```text theme={"system"}
https://model-<model_id>.api.baseten.co/environments/production/sync/v1
```

Use the model name from the deployment's example request in the Baseten dashboard, and your Baseten API key.

Because the URL points at the **production** environment, you can promote a new deployment to production in Baseten and your Bolna agents switch to it without a change.

***

## Finish on the agent

On the agent's **Intelligence** tab choose **custom**, pick the Baseten model, save and place a test call.

***

## Troubleshooting

> **401 Invalid or missing API key**
>
> Paste a key from **Settings → API keys** in the same Baseten workspace that owns the model or deployment.

> **Model not found on Model APIs**
>
> Use the slug exactly as listed, including the organisation prefix. A Fast variant has its own slug.

> **Slow first reply on a dedicated deployment**
>
> The deployment scaled down and is cold-starting. Set a minimum replica count of at least 1 in the deployment's autoscaling settings for live agents.

***

## Related

* [Custom LLM inference](https://www.bolna.ai/docs/custom-llm-inference)
* [Modal](https://www.bolna.ai/docs/custom-llm-inference/modal.md): write and run the serving code yourself
* [Fireworks AI](https://www.bolna.ai/docs/custom-llm-inference/fireworks.md): serverless models plus on-demand GPUs


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.