During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
Our GPU node pool stayed at eight nodes from Friday night to Monday morning with almost no GPU usage. The cluster autoscaler refused to remove them: notebook pods with local storage, a bare debug pod and a kube-system pod without a PDB. Fixing each blocker cut GPU spend by 38 percent.
Enabling dynamic batching in Triton Inference Server doubled throughput for our ranking model and pushed p99 latency from 38 to 85 ms, breaking our SLO. A large queue delay and preferred batch sizes made requests wait for batches that rarely filled at low traffic. Measured tuning fixed both.
Multi-camera computer vision at a stadium or arena cannot depend on the venue's uplink. Here is how we split work between an on-site GPU edge and the cloud, the bandwidth and latency math behind it, and the failure modes we design for: WAN loss, GPU faults, clock drift and bad models.
Ollama, vLLM and SGLang all expose an OpenAI-compatible API, so swapping them looks free. Under the hood they schedule requests and manage GPU memory very differently. We compare the queueing, batching and caching models and show how to measure which one fits your traffic.
We added a new GPU node pool and training pods stayed Pending with Insufficient nvidia.com/gpu, while the nodes sat idle. The NVIDIA device plugin was not running on the new nodes because its DaemonSet did not tolerate their taint, so the nodes never advertised GPUs.
Serving Llama 3.1 8B with vLLM on a 24 GB GPU crashed at startup with a KV cache error. By default vLLM sizes for the model's full 128k context, and after loading weights there was room for about 45k tokens of KV cache. Setting max_model_len to what we need fixed it.
Our self-hosted LLM box produced three tokens per second on a server with an RTX 4090. ollama ps reported 100% CPU: the container had no access to the GPU. Installing the NVIDIA Container Toolkit and requesting the GPU in Compose brought it to 120 tokens per second.
After rebuilding our inference image, models silently ran on CPU on half the GPU fleet. The new PyTorch wheel targets CUDA 12, and older nodes still had a driver that only supports CUDA 11. We aligned drivers, added a startup check that fails without a GPU, and pinned images.
Our text classification service crashed with CUDA out of memory every few hours, always on a big batch. Logits kept for drift monitoring were holding autograd graphs alive, batches were sized by count not tokens, and the allocator was fragmented. How we measured each one and fixed it.