Our team started with Ollama on a developer laptop, moved the same prompts to a shared GPU box for an internal assistant, and then watched latency fall apart once twenty people used it at the same time. The API had not changed, the model had not changed, and the GPU was barely half busy. What changed was the number of concurrent requests, and Ollama, vLLM and SGLang are built around very different assumptions about that number.

All three serve open-weight models behind an OpenAI-compatible HTTP API, which makes them look interchangeable. This post explains the core design of each engine (how requests are queued, how the GPU is shared between them, and what is cached), then gives a practical way to measure which one fits your traffic instead of trusting someone else's benchmark.

The question that decides everything

Before comparing features, write down three numbers for your workload:

  1. Concurrency: how many requests are in flight at peak. One developer is 1. A team chat tool is 5 to 50. A public product can be hundreds.
  2. Prefix overlap: how much of each prompt is shared with other prompts. A long system prompt plus tool schemas reused by every agent step means very high overlap. One-off summarisation of unrelated documents means almost none.
  3. Hardware: a laptop or CPU-only box, one consumer GPU, or data-centre GPUs with enough memory for unquantised weights.

Each engine is optimised for a different corner of that space.

Ollama: simple, local, one queue

Ollama wraps llama.cpp in a model manager and an HTTP server on port 11434. Models are pulled as GGUF files, usually already quantised to 4 or 5 bits, so a 7B or 8B model fits comfortably in laptop memory or a consumer GPU.

ollama pull llama3.1:8b
ollama serve
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1:8b", "messages": [{"role": "user", "content": "Say hi"}]}'

Request handling is deliberately simple. Each loaded model has a fixed number of parallel slots, controlled by OLLAMA_NUM_PARALLEL. Requests beyond that wait in a first-in, first-out queue (OLLAMA_MAX_QUEUE caps its length, and a full queue returns HTTP 503). Each slot reserves its own context, so raising parallelism multiplies the memory needed for context.

That design is excellent for one user: start-up is fast, the model unloads after OLLAMA_KEEP_ALIVE of idleness, CPU and Apple Silicon work out of the box, and there is no Python environment to manage. It is a poor fit for many concurrent users, because a long generation occupies its slot until it finishes and everyone behind it waits.

Use Ollama for local development, prototyping, offline work, and small tools with a handful of users.

vLLM: throughput under many concurrent requests

vLLM is built for serving many users from data-centre GPUs. Two techniques define it.

Continuous batching. A naive server collects a batch, runs it until every sequence finishes, then starts the next batch. Short answers sit idle while the longest one completes. vLLM schedules at the level of individual decoding steps: when a sequence finishes, a waiting request takes its place in the very next step. The GPU stays busy and short requests are not held hostage by long ones.

PagedAttention. Every request keeps a KV cache, the attention keys and values for tokens already processed, and it grows with every generated token. Allocating one contiguous block per request at its maximum length wastes most of the memory. vLLM stores the KV cache in fixed-size blocks, like virtual memory pages, and allocates them as sequences grow. Less waste means more concurrent sequences fit on the same GPU.

Since the V1 engine became the default, automatic prefix caching is on as well: KV blocks for identical prompt prefixes are reused across requests.

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128

The server listens on port 8000 by default. --gpu-memory-utilization sets how much of the GPU vLLM may claim for weights plus KV cache, --max-model-len caps context so the cache can hold more sequences, and --max-num-seqs limits how many sequences are batched at once.

The trade-offs: it expects NVIDIA or AMD data-centre class hardware for its best results, start-up takes longer because it profiles memory and captures CUDA graphs, and it reserves most of the GPU up front, so it does not share a card politely with other processes.

Use vLLM for production APIs with steady concurrency, batch generation jobs, and anywhere cost per token on GPUs matters.

SGLang: reuse across overlapping prompts

SGLang shares the serving fundamentals with vLLM (continuous batching, paged KV memory, an OpenAI-compatible server) and focuses on workloads where prompts overlap heavily.

Its central idea is RadixAttention. The KV cache for all recent requests is kept in a radix tree keyed by token sequences. A new request walks the tree, reuses the KV entries for the longest prefix it shares with anything cached, and only computes the remainder. Least-recently-used leaves are evicted when memory is needed. The scheduler is prefix-aware: it prefers to run requests that share cached prefixes together, which keeps hit rates high under load.

That fits agent loops very well. Every step of an agent resends the same system prompt, the same tool definitions and a growing conversation; only the newest turn is new. Multi-turn chat, few-shot classification with a fixed example block, and tree-style reasoning that branches from a shared prompt benefit the same way.

SGLang also has strong support for constrained decoding (JSON schema, regex and grammar-constrained output), which matters when an agent's tool calls must parse every time.

pip install "sglang[all]"
python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --port 30000 \
  --mem-fraction-static 0.85

The trade-offs are similar to vLLM: data-centre GPUs, a heavier install, and a faster-moving project where flags change between minor releases. Pin the version you tested.

Side by side

Aspect Ollama vLLM SGLang
Typical hardware Laptop, CPU, Apple Silicon, single consumer GPU Data-centre NVIDIA/AMD GPUs Data-centre NVIDIA/AMD GPUs
Model format GGUF, pre-quantised Hugging Face weights, AWQ/GPTQ/FP8 Hugging Face weights, AWQ/GPTQ/FP8
Scheduling Fixed slots + FIFO queue Continuous batching Continuous batching, prefix-aware
KV cache Per slot, sized for context Paged blocks + prefix caching Paged + radix tree reuse
Best at One user, easy setup High concurrency throughput Agents, multi-turn, structured output
Default port 11434 8000 30000

vLLM's prefix caching and SGLang's radix cache overlap in purpose. The difference shows up when overlap is high and the cache is under pressure: SGLang's tree and scheduler are designed around keeping shared prefixes resident, vLLM's hash-based block cache treats reuse as a bonus. For prompts with little overlap, the two tend to land close to each other and other factors (model support, deployment tooling, team familiarity) decide.

Measure with your own traffic shape

Published benchmarks rarely match your prompt lengths, output lengths or overlap. A one-hour test on your hardware is worth more. What we do:

  1. Export a few hundred real prompts (scrubbed of personal data) with their actual output lengths.
  2. Serve the same model at the same precision on each engine. Comparing a 4-bit GGUF in Ollama against FP16 in vLLM measures quantisation, not the engine.
  3. Replay the prompts at several concurrency levels: 1, 8, 32, 128.
  4. Record time to first token (TTFT), inter-token latency, and total output tokens per second at each level.

Both server projects ship load generators that speak their own and OpenAI-compatible APIs:

# vLLM benchmark client against any OpenAI-compatible server
vllm bench serve --backend openai-chat \
  --base-url http://localhost:8000 --endpoint /v1/chat/completions \
  --model Qwen/Qwen2.5-7B-Instruct \
  --dataset-name custom --dataset-path prompts.jsonl \
  --max-concurrency 32 --num-prompts 500

# SGLang benchmark client
python -m sglang.bench_serving --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --dataset-name random --random-input-len 2048 --random-output-len 256 \
  --num-prompts 500 --max-concurrency 32

Flag names differ between releases; check --help on the version you installed. For a fair comparison, drive every engine with the same client and the same dataset.

What the curves usually show: at concurrency 1, all three feel similar and Ollama's quick start wins for convenience. As concurrency rises, Ollama's TTFT climbs roughly in steps of whole generations because requests queue behind each other, while vLLM and SGLang keep TTFT flat much longer and total throughput keeps growing. With an agent-style dataset where most of each prompt is shared, SGLang's TTFT tends to stay lowest because most of the prefill is skipped.

A decision guide

Situation Start with
One developer, laptop, offline, quick experiments Ollama
Small internal tool, under five concurrent users, one consumer GPU Ollama, with OLLAMA_NUM_PARALLEL tuned
Public API or team-wide service, steady concurrency, mixed prompts vLLM
Agents, tool loops, long shared system prompts, multi-turn chat SGLang, or vLLM with prefix caching verified
Strict JSON or regex-shaped output at high volume SGLang or vLLM with structured outputs, measured

Whatever you choose, keep the client on the OpenAI-compatible API so you can switch engines later without rewriting the application. Only the base URL and model name should change.

Common mistakes

Mistake Why it hurts Do this instead
Benchmarking with concurrency 1 Hides the scheduling difference that matters in production Test at your peak concurrency and above
Comparing different quantisations You measure precision, not the engine Same weights and precision everywhere
Raising Ollama parallelism without checking memory Each slot reserves context; the model falls back to CPU Raise slowly, watch GPU memory and offload logs
Running vLLM next to other GPU processes It claims most of the GPU at start-up Dedicated GPU or lower --gpu-memory-utilization
Putting request-specific text at the start of prompts Breaks prefix reuse on vLLM and SGLang Keep shared content first, variable content last

Key takeaways

  • The three engines share an API but differ in scheduling: FIFO slots, continuous batching, and prefix-aware batching.
  • Ollama wins on simplicity and small hardware; it is not built for many concurrent users.
  • vLLM's continuous batching and paged KV cache maximise throughput under concurrency.
  • SGLang's radix cache shines when prompts share long prefixes, as in agents and multi-turn chat.
  • Decide with concurrency, prefix overlap and hardware, then confirm with a benchmark of your own prompts.

References