Our batch classification job with a small open model produced good output for the longest prompt in each batch and nonsense for the rest. Transformers warned that right-padding was detected for a decoder-only model. Left padding, a proper pad token and the attention mask fixed it.
Our text classification service crashed with CUDA out of memory every few hours, always on a big batch. Logits kept for drift monitoring were holding autograd graphs alive, batches were sized by count not tokens, and the allocator was fragmented. How we measured each one and fixed it.
Why does a long prompt make the first token slow while a long answer makes everything slow? A working model of tokens, prefill, decode and the KV cache, with the arithmetic we use to size GPUs and predict LLM latency and cost.