After a harmless-looking prompt change, our agent's time to first token went from 0.5 s to 1.6 s and GPU prefill load tripled. vLLM's prefix cache hit rate had dropped from 87% to 2%. The cause was one line at the top of the system prompt, plus tool schemas serialised in random order.
Serving Llama 3.1 8B with vLLM on a 24 GB GPU crashed at startup with a KV cache error. By default vLLM sizes for the model's full 128k context, and after loading weights there was room for about 45k tokens of KV cache. Setting max_model_len to what we need fixed it.
Why does a long prompt make the first token slow while a long answer makes everything slow? A working model of tokens, prefill, decode and the KV cache, with the arithmetic we use to size GPUs and predict LLM latency and cost.