vllm
[!CAUTION] This page and the related compose files were generated with AI (Claude Code) and have not been fully tested yet.
Container running vLLM to serve an open LLM model through an OpenAI-compatible API.
Compared to ollama, vLLM serves one model per container (no pull at runtime) and supports tool calling, which is useful to build and test LLM agents.
Requirements
An NVIDIA GPU is required with NVIDIA Container Toolkit (see also mborne.github.io/outils/cuda-toolkit - french).
[!NOTE] A CPU mode with vllm/vllm-openai-cpu was tried but is too slow for a dev environment (several minutes of
torch.compileat startup, slow generation with--enforce-eager). Use ollama without GPU.
Ensure that GPU support is enabled in docker :
docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark
Usage with docker
- Start :
docker compose up -d - Stop :
docker compose down
[!NOTE] The first start downloads the model from Hugging Face to the
vllmvolume and compiles it (torch.compile, cached in thevllm-compile-cachevolume). Usedocker compose logs -f vllmand wait forApplication startup complete(~2 minutes with the default model on a 6 GB GPU).
Configuration
Parameters are defined in .env :
| Variable | Default | Description |
|---|---|---|
VLLM_VERSION |
v0.33.0 |
Version of the vLLM image |
VLLM_PORT |
8000 |
Port exposed on the host |
VLLM_MODEL |
Qwen/Qwen3.5-0.8B |
Hugging Face model to serve |
VLLM_SERVED_MODEL_NAME |
qwen3.5-0.8b |
Model name to use in API requests (model field) |
VLLM_MAX_MODEL_LEN |
8192 |
Maximum context length (tokens) |
VLLM_MAX_NUM_SEQS |
16 |
Maximum number of concurrent requests |
VLLM_TOOL_CALL_PARSER |
qwen3_coder |
Tool call parser matching VLLM_MODEL |
VLLM_GPU_MEMORY_UTILIZATION |
0.75 |
Fraction of the GPU memory used by vLLM |
HF_TOKEN |
(empty) | Hugging Face token, required only for gated models |
[!WARNING] When changing
VLLM_MODEL, check thevllm servecommand given in the model card and adaptVLLM_TOOL_CALL_PARSER(see vLLM - Tool calling).
Choosing a model
Last update : 2026-11-11
The default model (Qwen/Qwen3.5-0.8B, ~1.8 GB in BF16) is chosen to start on small GPUs (tested with 6 GB of VRAM : ~4.1 GB used). Models from the same family can be used depending on the available VRAM (BF16 weights, add some memory for the KV cache) :
| Model | Weights | Target |
|---|---|---|
| Qwen/Qwen3.5-0.8B | ~1.8 GB | GPU with 6+ GB of VRAM |
| Qwen/Qwen3.5-2B | ~4.6 GB | GPU with 8+ GB of VRAM |
| Qwen/Qwen3.5-4B | ~9.3 GB | GPU with 12+ GB of VRAM |
| Qwen/Qwen3.5-9B | ~19 GB | GPU with 24+ GB of VRAM |
[!NOTE] Qwen3.5 models are multimodal (text and images). Qwen3.5-0.8B and 2B answer without thinking by default, while 4B and above think by default : add
--reasoning-parser=qwen3to thecommandto get the reasoning in a separatereasoning_contentfield.
API usage
Use these commands once vLLM is started :
# list models
curl -sS http://localhost:8000/v1/models | jq -r '.data[].id'
# chat completion
curl -sS http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.5-0.8b",
"messages": [{"role": "user", "content": "Why is the sky blue?"}]
}' | jq -r '.choices[0].message.content'
The API is also available through traefik at https://vllm.dev.localhost/v1/models.
Any OpenAI client can be used, for instance with Python :
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
response = client.chat.completions.create(
model="qwen3.5-0.8b",
messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(response.choices[0].message.content)
Troubleshooting
- Out of memory at startup : reduce
VLLM_MAX_MODEL_LEN(ex :4096) orVLLM_GPU_MEMORY_UTILIZATION, or use a smaller model. max_num_seqs (...) exceeds available Mamba cache blocks (...): models with linear attention layers such as Qwen3.5 reserve one cache block per concurrent request. LowerVLLM_MAX_NUM_SEQSor increaseVLLM_GPU_MEMORY_UTILIZATION.could not select device driver "nvidia": GPU support is not enabled in docker (see mborne.github.io/outils/cuda-toolkit).404withThe model ... does not exist: themodelfield must matchVLLM_SERVED_MODEL_NAME, not the Hugging Face model name.
Resources
vLLM :
- docs.vllm.ai - Using Docker
- docs.vllm.ai - OpenAI-compatible server
- docs.vllm.ai - Tool calling
- hub.docker.com - vllm/vllm-openai
GPU support :
Clients :