We built an internal assistant that researches crypto markets and drafts a briefing for analysts, using a planner and specialised sub-agents. The lessons: isolating sub-agent context, a file scratchpad instead of one huge conversation, cost and tool caps, and evals. It does research, never trading.
Our support agent started opening refund requests when customers only asked where their order was. Tool names and descriptions overlapped, and the agent had 23 tools to choose from. Clear tool contracts, fewer tools per step and a tool-choice eval fixed it.
A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.
Our internal assistant answered questions about supplier contracts and price lists, and its answers about prices were confidently wrong. Plain text extraction had flattened PDF tables into run-on text. How we found it, switched to layout-aware parsing with tables as chunks, and measured the fix.
Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.