> ## Documentation Index > Fetch the complete documentation index at: https://www.bolna.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Self-host an LLM on Modal for Bolna voice agents > Deploy an open-weight model with vLLM on Modal serverless GPUs, protect it with a proxy token, keep it warm, and use it as a Bolna custom LLM. [Modal](https://modal.com) is a serverless GPU platform: you write a small Python app, run `modal deploy`, and Modal gives you a public HTTPS URL backed by GPUs. Unlike the other providers in this section, Modal has no model catalog and no shared endpoint. **You run the inference server.** Choose Modal when you want a fine-tuned checkpoint, a model no hosted API offers, a GPU region close to your callers, or full control over the serving engine. This guide deploys an OpenAI-compatible [vLLM](https://docs.vllm.ai) server on Modal and adds it to Bolna as a [custom LLM](https://www.bolna.ai/docs/custom-llm-inference). ## What to enter in Bolna | Field | Value | | - | - | | **Provider** | Modal | | **Model name** | A value you gave vLLM's `--served-model-name`, for example `llm` | | **Endpoint URL** | The URL `modal deploy` prints, followed by `/v1` | | **API key** | A Modal proxy token, written as `wk-.ws-` | | **Family** | The model family, for example `gemma` or `qwen` | *** ## What you need * A Modal account and the Modal CLI (`pip install modal`, then `modal setup`) * The Hugging Face model you want to serve * A Bolna account with access to **Settings → Integrations** *** ## Deploy and connect **1. Deploy a vLLM server** Save this as `bolna_llm.py` and run `modal deploy bolna_llm.py`. It is a trimmed version of Modal's [vLLM inference example](https://modal.com/docs/examples/vllm_inference), with the settings a voice agent needs. Swap in your own model and the GPU it needs. ```python bolna_llm.py theme={"system"} import json import subprocess import modal MODEL = "google/gemma-4-26B-A4B-it" image = ( modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.12") .entrypoint([]) .uv_pip_install("vllm==0.21.0") .env({"HF_XET_HIGH_PERFORMANCE": "1"}) ) app = modal.App("bolna-llm") @app.server( image=image, gpu="H200", port=8000, startup_timeout=10 * 60, min_containers=1, # a Server with no running container rejects requests target_concurrency=100, # requests per container before Modal adds another routing_region="ap-south", # Mumbai; pick the region closest to your callers volumes={ "/root/.cache/huggingface": modal.Volume.from_name("huggingface-cache", create_if_missing=True), "/root/.cache/vllm": modal.Volume.from_name("vllm-cache", create_if_missing=True), }, ) class Server: @modal.enter() def start(self): self.process = subprocess.Popen([ "vllm", "serve", MODEL, "--served-model-name", "llm", "--host", "0.0.0.0", "--port", "8000", "--limit-mm-per-prompt", json.dumps({"image": 0, "video": 0, "audio": 0}), "--enable-auto-tool-choice", "--tool-call-parser", "gemma4", ]) @modal.exit() def stop(self): self.process.terminate() ``` `modal deploy` prints the server URL, for example `https://your-workspace--bolna-llm-server.ap-south.modal.direct`. The dialog's placeholder shows the older `*.modal.run` shape that Modal web functions use; both work the same way. Paste the URL you were given and add `/v1`. **2. Create a proxy token** A Modal Server rejects requests without a proxy token by default, with a `401`. Create one under [Settings → Proxy Auth Tokens](https://modal.com/settings/proxy-auth-tokens) in Modal, or run `modal workspace proxy-tokens create`. Join the token ID and secret with a period, `wk-.ws-`, and use that as the API key. Modal accepts it as a bearer token, which is what Bolna sends. If your workspace uses role-based access, scope the token to the environment the app is deployed in, or requests are denied with a `403`. **3. Check the endpoint** ```bash theme={"system"} curl https:///v1/models \ -H "Authorization: Bearer wk-.ws-" ``` The `id` in the response is the model name to register, `llm` in the example above. To also measure time to first token, run the [streaming test](https://www.bolna.ai/docs/custom-llm-inference#test-your-endpoint-before-you-register-it). **4. Choose Modal in the Add custom model dialog** In Bolna, go to **Settings → Integrations**, filter by **LLM**, and click **Add model** on the **Custom LLM** card. You can also open the same dialog from an agent: on the **Intelligence** tab, open the **LLM Provider** dropdown and choose **+ Add custom LLM**. Pick **Modal** as the provider. The endpoint field only shows the shape of a Modal URL as a placeholder, because your URL is unique to your workspace and app. ![Bolna Add custom model dialog with Modal selected as provider and an empty endpoint URL field showing a Modal URL placeholder](https://mintcdn.com/bolna-54a2d4fe/0EXypen329Eb5XE2/images/custom-llm/providers/modal-selected.png?fit=max\&auto=format\&n=0EXypen329Eb5XE2\&q=85\&s=2b800720a839256685b7cbb1ef20ef63) Modal selected: the endpoint placeholder shows the URL shape, the hint says to use the URL from modal deploy **5. Register the model** **Dashboard** Fill in the values from [the table above](#what-to-enter-in-bolna) and click **Add model**. ![Add custom model dialog with Modal selected, a model name, a Modal endpoint URL, a masked API key and a family](https://mintcdn.com/bolna-54a2d4fe/aV72vktown5Rn3f1/images/custom-llm/providers/modal-filled.png?fit=max\&auto=format\&n=aV72vktown5Rn3f1\&q=85\&s=c694afc737e83456ca3817d180d59c5f) A filled-in Modal registration **API** ```bash theme={"system"} curl --request POST \ --url https://api.bolna.ai/user/model/custom \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "custom_model_name": "llm", "custom_model_url": "https:///v1", "custom_model_key": "wk-.ws-", "custom_model_family": "gemma" }' ``` **6. Use it on an agent** On the agent's **Intelligence** tab, choose **custom** as the provider and select your model. If you added it from the agent's dropdown, it is already selected. Save the agent and place a web call before you send live traffic to it. *** ## Things to know * **Keep a container warm.** A Modal Server with no running container rejects requests with a `503` instead of queuing them, so a call that arrives after the app has scaled to zero gets no reply at all. Keep `min_containers=1` for any agent that takes live calls. You pay for that GPU while it idles. * **Region.** `routing_region` sets where Modal's proxy accepts requests: `us-east` (default), `us-west`, `ca-central`, `eu-west`, `ap-south` (Mumbai) or `ap-southeast-2`. To also keep the GPUs in that region, set `compute_region`, which adds a [cost multiplier](https://modal.com/docs/guide/region-selection#pricing). * **Model name.** Bolna checks the model name against your server's `/v1/models` when you add it. vLLM lists every value given to `--served-model-name`, so enter one of them exactly. * **Tool calling.** If your agent uses tools, vLLM needs `--enable-auto-tool-choice` and the parser for your model, for example `gemma4` for Gemma 4, `hermes` for Qwen2.5 or `llama3_json` for Llama 3.1. See vLLM's [tool calling guide](https://docs.vllm.ai/en/latest/features/tool_calling/). * **Thinking.** Qwen3 models think before every reply by default, which the caller hears as silence. Turn it off per agent with the [extra request params](https://www.bolna.ai/docs/custom-llm-inference#extra-request-params) `{"chat_template_kwargs": {"enable_thinking": false}}`. Gemma 4 has thinking off by default. * **Using vLLM's own key instead.** If you deploy with `unauthenticated=True`, Modal's proxy lets every request through. In that case, protect the server with vLLM's `--api-key` flag, read from a [Modal Secret](https://modal.com/docs/guide/secrets), and register that key in Bolna instead of a proxy token. ## Tuning a Modal deployment for voice | Setting | Where | Why it matters on a call | | - | - | - | | `min_containers` | `@app.server` | Keep at least `1`; with none running, requests get a `503` | | `target_concurrency` | `@app.server` | Concurrent requests per container before Modal adds another; size it to your peak concurrent calls | | `routing_region` | `@app.server` | Put the endpoint close to your callers to cut network time on every turn | | GPU type | `gpu=` | Larger GPUs cut time to first token for bigger models | | `--max-model-len` | vLLM flags | Smaller context windows boot faster and leave more memory for concurrent calls | *** ## Troubleshooting > **Calls get no reply, or the first call after a quiet period fails** > > The app scaled to zero, and Modal rejected the request with a `503`. Registration still saves a model it can't reach, so this first shows up on a call. Set `min_containers=1` and redeploy. > **The endpoint rejected this API key** > > Check that the key is the token ID and secret joined with a period, `wk-.ws-`, with no spaces. A `403` means the token isn't scoped to the app's environment. If you use vLLM's `--api-key` instead, make sure the Secret is attached and redeploy after changing it. > **Model not found** > > The model name must match a `--served-model-name` value exactly. Check `GET /v1/models`. > **404 on chat completions** > > The URL is missing `/v1`, or it points at a different Modal app. Use the URL that `modal deploy` printed for the server, followed by `/v1`. *** ## Related * [Custom LLM inference](https://www.bolna.ai/docs/custom-llm-inference): every field, the save-time checks and extra request params * [Modal vLLM inference example](https://modal.com/docs/examples/vllm_inference), [Modal Servers](https://modal.com/docs/guide/servers) and [proxy auth tokens](https://modal.com/docs/guide/webhook-proxy-auth) * [Baseten](https://www.bolna.ai/docs/custom-llm-inference/baseten.md): managed dedicated deployments if you'd rather not maintain the server code This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.