Custom LLMs for Bolna Voice Agents

Bring your own model by pointing Bolna at any endpoint that speaks the OpenAI chat completions API. Your endpoint can run anywhere: vLLM, Ollama, SGLang, TGI, a LiteLLM proxy, or a hosted gateway. Bolna calls it exactly as it calls OpenAI, only at your URL and with your key. Use this when you need a fine-tuned or open-weight model, when the weights have to stay on your own infrastructure, or when you already pay a provider directly and want the tokens billed there.

Quick config

"llm_agent": {
  "agent_type": "simple_llm_agent",
  "agent_flow_type": "streaming",
  "llm_config": {
    "provider": "custom",
    "model": "Qwen/Qwen2.5-3B-Instruct",
    "base_url": "https://your-host/v1",
    "family": "qwen",
    "max_tokens": 150,
    "temperature": 0.2
  }
}

Register the model once under Integrations → LLM → Custom LLM in the dashboard, then select it on the agent’s LLM tab. See Using custom LLMs for the walkthrough and the API.

What your endpoint must support

Bolna sends requests with the OpenAI SDK, so the endpoint has to behave like OpenAI’s chat completions API.

Requirement	Detail
`POST {base_url}/chat/completions`	Give Bolna the base URL including the version segment, for example `https://your-host/v1`. Bolna appends the rest of the path.
Streaming responses	OpenAI delta format, terminated by `data: [DONE]`. Tokens stream straight into speech synthesis, so an endpoint that only replies once the full answer is ready adds audible silence before the agent speaks.
Bearer authentication	Your key is sent in the `Authorization` header.
A publicly resolvable host	Private, loopback and link-local addresses are refused, both at registration and on every call.
Streamed `tool_calls`	Only if your agent uses tools. Support varies between self-hosted servers, so test it before relying on it.

Your model id is passed through unchanged, so it must match whatever your server expects.

Key settings

Setting	Type	Recommended	Description
`provider`	string	`"custom"`	Provider name
`model`	string	e.g. `"Qwen/Qwen2.5-3B-Instruct"`	The model id your endpoint serves, passed through as-is
`base_url`	string	e.g. `"https://your-host/v1"`	Your OpenAI-compatible base URL, including the version segment
`family`	string	e.g. `"qwen"`	Model family, used for prompt formatting hints
`max_tokens`	integer	`150`	Cap on response length, keep it short for voice
`temperature`	float	`0.2`	Lower is more consistent
`agent_flow_type`	string	`"streaming"`	Always `"streaming"` for voice

When to use a custom LLM vs a hosted provider

Use a custom LLM when:

You run a fine-tuned or open-weight model that no hosted provider offers
The weights or the traffic have to stay on your own infrastructure
You already have a contract with a provider and want the tokens billed to it directly

Use a hosted provider integration when:

You want a frontier model without operating an inference server
You need Bolna’s per-provider features, such as Azure data residency or provisioned throughput
You would rather not own capacity planning, cold starts and upgrades

FAQ

Does Bolna use the OpenAI Responses API for custom models?

No. Custom models always use chat completions, because the Responses API’s server-side conversation chaining is specific to OpenAI and a self-hosted server cannot resolve a previous response id.

What happens if my endpoint has scaled to zero?

The first call pays the cold start, which the caller hears as silence before the agent speaks. Keep a warm instance if first-token latency matters. Registration tolerates an unreachable endpoint, so scaling to zero does not block you from adding the model.

Who bills me for the tokens?

Your provider. Bolna records a custom model’s token usage against your own endpoint, so that model’s inference is not billed by Bolna.

Can a sub-account have its own custom model?

Yes. A model added on the account is available to it and to all of its sub-accounts. A model added inside a sub-account stays in that sub-account, invisible to the main account and to sibling sub-accounts.

Can I point an agent at a model on a private network?

No. The endpoint must resolve to a public address. Expose it through a public hostname, with your key as the bearer token.

Using custom LLMs: register, validate, manage and delete a custom model
LLM Tab: configure LLM in the dashboard
OpenAI: direct OpenAI integration
OpenRouter: one key for many hosted providers
Prompting Guide: write effective prompts for voice

Overview & Evaluation

Concepts

Build & Configure Agents

Voice & Knowledge

Workflows

Integrations

Multilingual Voice Agents

Advanced Capabilities

Graph Agents

Enterprise & Security

Pricing

Custom LLMs for Bolna Voice Agents

Quick config

What your endpoint must support

Key settings

When to use a custom LLM vs a hosted provider

FAQ

​Quick config

​What your endpoint must support

​Key settings

​When to use a custom LLM vs a hosted provider

​FAQ

​Related

Quick config

What your endpoint must support

Key settings

When to use a custom LLM vs a hosted provider

FAQ

Related