Model training/serving bugs, LLM integrations, GPU/CUDA issues.
Our internal assistant answered questions about supplier contracts and price lists, and its answers about prices were confidently wrong. Plain text extraction had flattened PDF tables into run-on text. How we found it, switched to layout-aware parsing with tables as chunks, and measured the fix.
Prompt and response logs are the fastest way to build an accidental PII database. This guide shows where personal data enters an LLM feature, how to redact it before logging and before the provider, and how to keep retention and access under control.
We added a semantic cache to our help assistant to cut LLM cost. Within days a user asking how to cancel a subscription got instructions for cancelling an order. The two questions were 0.92 similar. We added exact-match first, scoped keys, intent gating and a measured threshold.
Ollama, vLLM and SGLang all expose an OpenAI-compatible API, so swapping them looks free. Under the hood they schedule requests and manage GPU memory very differently. We compare the queueing, batching and caching models and show how to measure which one fits your traffic.
Our docs assistant answered v4 questions with v3 settings, fluently and with citations. Fixed-size chunking split tables from their headings, chunks carried no version metadata and we had no retrieval eval. How we rebuilt chunking, added filters and hybrid search, and measured it.
A summarisation service trimmed documents to fit a token budget, yet some requests failed with a context length error. Our counter used a tokenizer from a different model family, ignored the chat template, tool schemas and the reserved output tokens. How we found the gap and switched to the provider's token counting.
After we turned on streaming for our assistant, tool calls failed with JSON parse errors and sometimes ran with another call's arguments. We were parsing argument fragments per chunk and keying them by arrival order. Accumulating by index fixed it.
A research agent called the same search tool 412 times in one run and spent more overnight than the whole feature spent in a normal week. The tool returned an empty string on errors, the loop had no step limit and nothing tracked spend per run.
Our batch classification job with a small open model produced good output for the longest prompt in each batch and nonsense for the rest. Transformers warned that right-padding was detected for a decoder-only model. Left padding, a proper pad token and the attention mask fixed it.
A coding agent implemented a discount rule and wrote tests for it. Everything was green, the review looked fine, and production applied the discount to the wrong orders. The agent had written tests from its own implementation, so they asserted the bug. How we changed the workflow so humans own the acceptance criteria.
About 2% of our LLM extraction calls crashed on JSON.parse: markdown code fences, answers cut off at max_tokens and chatty text after the object. We classified every failure, switched to schema-constrained output, checked stop reasons and added validation with one repair attempt.
Our first voice assistant prototype took two to three seconds to answer, and callers talked over it. Most of the delay was not the language model. How we built a latency budget across audio transport, end-of-turn detection, speech recognition, the LLM and speech synthesis, and what we changed in each stage.
Tenants upload terabytes of audio, video, PDFs and images, and expect an LLM to answer questions over all of it, with their data isolated and a bill that matches what they used. Here is the AWS design we shipped: ingestion pipeline, isolation layers and metering.
Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.
A pull request added a dependency that an AI assistant had suggested. The package name did not exist when the code was written, and a few weeks later someone else published it. We caught it in review by luck, then added registry checks so we do not rely on luck.
Our internal MCP server passed every test in the inspector but kept failing in the desktop client with an 'is not valid JSON' error. A startup banner and a debug print went to stdout, which is the protocol channel in stdio transport. Logging to stderr fixed it.
Our document tagger hit HTTP 429 constantly while sending only 40 requests a minute, far below the request limit. The token-per-minute limit was the real ceiling, inflated by an oversized max_tokens and minute-aligned bursts. How we read the headers and built a client-side limiter.
Our meeting transcripts contained sentences nobody said, usually during silence or hold music, plus the same line repeated a dozen times. Whisper conditions on its own previous output and has no idea what silence is. A VAD pre-filter and stricter decoding thresholds removed most of it.
We added cache_control to a 6,000-token system prompt and the bill went up instead of down. Every request wrote the cache and none read it, because the prompt prefix changed on every call. A stable prefix brought cache reads to 92 percent.
Our chat assistant streamed tokens smoothly on localhost but in production the whole answer appeared after 14 seconds. nginx was buffering the event stream and gzip was holding bytes. How we proved it with curl and fixed it without breaking other routes.
Serving Llama 3.1 8B with vLLM on a 24 GB GPU crashed at startup with a KV cache error. By default vLLM sizes for the model's full 128k context, and after loading weights there was room for about 45k tokens of KV cache. Setting max_model_len to what we need fixed it.
Our RAG prototype was one folder where the PDF loader, the vector client and the API route imported each other freely. Here is the layout we moved to: four packages with enforced boundaries, a shared contract module, versioned indexes and an eval gate in CI.
Small tenants in our RAG product got 2 or 3 chunks back instead of 10, and sometimes none. HNSW finds ef_search candidates first and applies the tenant filter afterwards. Iterative index scans in pgvector 0.8 plus partial indexes for big tenants fixed recall.
Our self-hosted LLM box produced three tokens per second on a server with an RTX 4090. ollama ps reported 100% CPU: the container had no access to the GPU. Installing the NVIDIA Container Toolkit and requesting the GPU in Compose brought it to 120 tokens per second.
Semantic search in our docs assistant took 1.9 seconds and returned odd rankings. The query sorted by L2 distance while the HNSW index was built for cosine, so Postgres scanned every row. Matching the operator to the opclass brought it to 14 ms.