Our support agent started opening refund requests when customers only asked where their order was. Tool names and descriptions overlapped, and the agent had 23 tools to choose from. Clear tool contracts, fewer tools per step and a tool-choice eval fixed it.
When an agent answer is slow or wrong, logs rarely tell you which step caused it. This guide shows how to trace LLM calls, tool executions and agent runs with OpenTelemetry's GenAI semantic conventions, which attributes to record, and how to handle prompts and privacy.
After a harmless-looking prompt change, our agent's time to first token went from 0.5 s to 1.6 s and GPU prefill load tripled. vLLM's prefix cache hit rate had dropped from 87% to 2%. The cause was one line at the top of the system prompt, plus tool schemas serialised in random order.
In a red-team exercise our support agent read a customer-uploaded PDF with hidden instructions and drafted an email with another customer's order details. No data left the building, but the flaw was real. How we contained it with least privilege, confirmation and output checks.
A research agent called the same search tool 412 times in one run and spent more overnight than the whole feature spent in a normal week. The tool returned an empty string on errors, the loop had no step limit and nothing tracked spend per run.
Our internal MCP server passed every test in the inspector but kept failing in the desktop client with an 'is not valid JSON' error. A startup banner and a debug print went to stdout, which is the protocol channel in stdio transport. Logging to stderr fixed it.
Our support agent started failing with a 400 on roughly one conversation in twelve. The model had begun returning several tool calls in one turn and our loop only answered the first. Answering every tool_call_id in order fixed it.