Serving LLMs on Kubernetes: KServe vs KubeRay with Ray Serve for vLLM inference
Both KServe and KubeRay can put vLLM behind an OpenAI-compatible endpoint on Kubernetes. They differ in who owns the control plane, how scaling decisions are made and how multi-node models are handled. We ran both for the same model and this is how we chose.