We built an internal assistant that researches crypto markets and drafts a briefing for analysts, using a planner and specialised sub-agents. The lessons: isolating sub-agent context, a file scratchpad instead of one huge conversation, cost and tool caps, and evals. It does research, never trading.
We built a pre-push gate that runs linters, secret and SAST scans, containerised tests and an LLM review of the diff before code leaves a laptop. The hard parts were not the model call: ordering, cost caps, redaction, fail-open rules and keeping CI as the real authority.
Teams often reach for fine-tuning when they need retrieval, or build a retrieval pipeline when the whole corpus fits in the prompt. This guide compares RAG, fine-tuning and long context on freshness, cost, latency, citations and effort, with a decision table and examples.
Our support agent started opening refund requests when customers only asked where their order was. Tool names and descriptions overlapped, and the agent had 23 tools to choose from. Clear tool contracts, fewer tools per step and a tool-choice eval fixed it.
Users of our internal assistant kept repeating the same preferences: their team, report format, time zone. Adding memory sounds simple and is not. How we separated short-term and long-term memory, decided what to store, handled conflicts and stale facts, and gave users control over what the assistant remembers.
Our expense app sends receipt photos to a vision model. The feature cost six times our estimate and p95 latency was 9 seconds. Phones upload 12-megapixel photos and we sent them as-is. Resizing and cropping before upload cut cost by 78 percent with no accuracy loss.
When an agent answer is slow or wrong, logs rarely tell you which step caused it. This guide shows how to trace LLM calls, tool executions and agent runs with OpenTelemetry's GenAI semantic conventions, which attributes to record, and how to handle prompts and privacy.
After a harmless-looking prompt change, our agent's time to first token went from 0.5 s to 1.6 s and GPU prefill load tripled. vLLM's prefix cache hit rate had dropped from 87% to 2%. The cause was one line at the top of the system prompt, plus tool schemas serialised in random order.
Our LLM bill tripled in one month and nobody could say which feature caused it. This guide covers what we built since: per-feature token accounting, prompt and response caching, model routing, output limits, batch processing and budgets with alerts.
Connecting WhatsApp, Telegram or Slack to an LLM agent that can run tools on your server is a few hundred lines of glue and a large attack surface. This is the architecture we use: normalised channel adapters, sender pairing, a locked-down tool sandbox and hard budget caps.
Once more than two services call LLM providers, each one reinvents timeouts, retries and key handling, usually badly. This guide covers what a small internal LLM gateway should do, the timeout and retry rules that work, and how to fail over without making outages worse.
In a red-team exercise our support agent read a customer-uploaded PDF with hidden instructions and drafted an email with another customer's order details. No data left the building, but the flaw was real. How we contained it with least privilege, confirmation and output checks.
Most agent demos are one big loop around a model. In production we replaced ours with four explicit patterns: a router, a planner with a fixed step budget, a bounded reflection pass and an approval gate for side effects. Here is how each works and where it fails.
Our nightly product description job used the Batch API to save cost. One night the batch expired with 30 percent unfinished, and our importer joined results by line position, so descriptions landed on the wrong products. We now join by custom_id and handle every terminal status.
A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.
Prompt and response logs are the fastest way to build an accidental PII database. This guide shows where personal data enters an LLM feature, how to redact it before logging and before the provider, and how to keep retention and access under control.
We added a semantic cache to our help assistant to cut LLM cost. Within days a user asking how to cancel a subscription got instructions for cancelling an order. The two questions were 0.92 similar. We added exact-match first, scoped keys, intent gating and a measured threshold.
Ollama, vLLM and SGLang all expose an OpenAI-compatible API, so swapping them looks free. Under the hood they schedule requests and manage GPU memory very differently. We compare the queueing, batching and caching models and show how to measure which one fits your traffic.
A summarisation service trimmed documents to fit a token budget, yet some requests failed with a context length error. Our counter used a tokenizer from a different model family, ignored the chat template, tool schemas and the reserved output tokens. How we found the gap and switched to the provider's token counting.
A research agent called the same search tool 412 times in one run and spent more overnight than the whole feature spent in a normal week. The tool returned an empty string on errors, the loop had no step limit and nothing tracked spend per run.
Our batch classification job with a small open model produced good output for the longest prompt in each batch and nonsense for the rest. Transformers warned that right-padding was detected for a decoder-only model. Left padding, a proper pad token and the attention mask fixed it.
About 2% of our LLM extraction calls crashed on JSON.parse: markdown code fences, answers cut off at max_tokens and chatty text after the object. We classified every failure, switched to schema-constrained output, checked stop reasons and added validation with one repair attempt.
Tenants upload terabytes of audio, video, PDFs and images, and expect an LLM to answer questions over all of it, with their data isolated and a bill that matches what they used. Here is the AWS design we shipped: ingestion pipeline, isolation layers and metering.
Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.
Our internal MCP server passed every test in the inspector but kept failing in the desktop client with an 'is not valid JSON' error. A startup banner and a debug print went to stdout, which is the protocol channel in stdio transport. Logging to stderr fixed it.