Ollama, vLLM and SGLang all expose an OpenAI-compatible API, so swapping them looks free. Under the hood they schedule requests and manage GPU memory very differently. We compare the queueing, batching and caching models and show how to measure which one fits your traffic.
Our text classification service crashed with CUDA out of memory every few hours, always on a big batch. Logits kept for drift monitoring were holding autograd graphs alive, batches were sized by count not tokens, and the allocator was fragmented. How we measured each one and fixed it.
Why does a long prompt make the first token slow while a long answer makes everything slow? A working model of tokens, prefill, decode and the KV cache, with the arithmetic we use to size GPUs and predict LLM latency and cost.