Tag See all →

#llm-serving

2 stories

Banner: two Kubernetes control planes side by side routing requests to GPU nodes running an inference engine

DevOps & CI/CD

Serving LLMs on Kubernetes: KServe vs KubeRay with Ray Serve for vLLM inference

John Nguyen · Dec 19, 2025 · 9 min read

Both KServe and KubeRay can put vLLM behind an OpenAI-compatible endpoint on Kubernetes. They differ in who owns the control plane, how scaling decisions are made and how multi-node models are handled. We ran both for the same model and this is how we chose.

Banner: vLLM startup failure because the configured context length does not fit in the KV cache

AI & Machine Learning Bug Report

vLLM fails to start: max seq len is larger than the tokens the KV cache can store

Ryan Pham · Mar 26, 2025 · 5 min read

Serving Llama 3.1 8B with vLLM on a 24 GB GPU crashed at startup with a KV cache error. By default vLLM sizes for the model's full 128k context, and after loading weights there was room for about 45k tokens of KV cache. Setting max_model_len to what we need fixed it.