Our vLLM deployment had an HPA targeting 70 percent CPU. Under a load spike, requests queued for 40 seconds while CPU sat at 15 percent, so nothing scaled. GPU-bound servers need to scale on queue depth. We switched to KEDA on vLLM's waiting-requests metric and fixed warm-up.
After a harmless-looking prompt change, our agent's time to first token went from 0.5 s to 1.6 s and GPU prefill load tripled. vLLM's prefix cache hit rate had dropped from 87% to 2%. The cause was one line at the top of the system prompt, plus tool schemas serialised in random order.
Both KServe and KubeRay can put vLLM behind an OpenAI-compatible endpoint on Kubernetes. They differ in who owns the control plane, how scaling decisions are made and how multi-node models are handled. We ran both for the same model and this is how we chose.
Ollama, vLLM and SGLang all expose an OpenAI-compatible API, so swapping them looks free. Under the hood they schedule requests and manage GPU memory very differently. We compare the queueing, batching and caching models and show how to measure which one fits your traffic.
Serving Llama 3.1 8B with vLLM on a 24 GB GPU crashed at startup with a KV cache error. By default vLLM sizes for the model's full 128k context, and after loading weights there was room for about 45k tokens of KV cache. Setting max_model_len to what we need fixed it.
Why does a long prompt make the first token slow while a long answer makes everything slow? A working model of tokens, prefill, decode and the KV cache, with the arithmetic we use to size GPUs and predict LLM latency and cost.