A research agent called the same search tool 412 times in one run and spent more overnight than the whole feature spent in a normal week. The tool returned an empty string on errors, the loop had no step limit and nothing tracked spend per run.
Every value in our orders API was a bound parameter, except the sort column, which was pasted into ORDER BY from the query string. A pentest proved it was injectable. We replaced it with an allowlist map and added tests that fuzz the sort input.
Our monthly statement job used a Redis SET NX lock with a 30 second TTL. One slow run outlived the TTL, a second worker took the lock, and 214 customers got duplicated statement lines. Here is why a TTL lock alone cannot be safe and how fencing tokens closed the gap.
Our batch classification job with a small open model produced good output for the longest prompt in each batch and nonsense for the rest. Transformers warned that right-padding was detected for a decoder-only model. Left padding, a proper pad token and the attention mask fixed it.
One slow partner API call made our whole FastAPI service stop answering, health checks included, until Kubernetes restarted the pods. A synchronous requests call and a time.sleep inside async def were blocking the event loop.
About 2% of our LLM extraction calls crashed on JSON.parse: markdown code fences, answers cut off at max_tokens and chatty text after the object. We classified every failure, switched to schema-constrained output, checked stop reasons and added validation with one repair attempt.
A small refactor added __eq__ to our Money class so amounts could be compared. Soon a cache decorator and several set operations failed with TypeError: unhashable type: 'Money', because defining __eq__ without __hash__ sets __hash__ to None. How equality and hashing must agree.
Customer CSV uploads failed with KeyError 'email' even though the header clearly said email. Excel had written a UTF-8 byte order mark, and other files arrived in Windows-1252. We now detect the encoding explicitly and normalise headers.
Our bot believed two orders were still open when they had filled hours earlier. The exchange closes websocket connections after 24 hours; our reconnect logic resubscribed but never checked what happened while disconnected. Reconnect with backoff plus REST reconciliation fixed the state drift.
A pull request added a dependency that an AI assistant had suggested. The package name did not exist when the code was written, and a few weeks later someone else published it. We caught it in review by luck, then added registry checks so we do not rely on luck.
A Lambda function that converts uploaded videos to thumbnails worked for the first few invocations, then failed with No space left on device. Files written to /tmp survived across warm invocations of the same execution environment and filled the 512 MB default. How we found it and fixed cleanup, sizing and streaming.
Accuracy is the metric everyone wants, and for many models the labels arrive weeks or months after the prediction. This guide covers what to monitor in the meantime: data quality, input drift with PSI and KS tests, prediction drift, and alert design that does not cry wolf.
Our internal MCP server passed every test in the inspector but kept failing in the desktop client with an 'is not valid JSON' error. A startup banner and a debug print went to stdout, which is the protocol channel in stdio transport. Logging to stderr fixed it.
Our order router ran fine as one process. When we scaled to two workers, a third of private API calls failed with EAPI:Invalid nonce. Kraken requires an increasing nonce per API key, and two processes generating millisecond timestamps raced each other. One key per worker fixed it.
Our document tagger hit HTTP 429 constantly while sending only 40 requests a minute, far below the request limit. The token-per-minute limit was the real ceiling, inflated by an oversized max_tokens and minute-aligned bursts. How we read the headers and built a client-side limiter.
Our deploy tool crashed with 'bool' object has no attribute 'upper' the day we added Norway to a YAML list. YAML 1.1 implicit typing turned NO into false and 0755 into 493. Here is how we found it, fixed it and stopped it coming back.
Our meeting transcripts contained sentences nobody said, usually during silence or hold music, plus the same line repeated a dozen times. Whisper conditions on its own previous output and has no idea what silence is. A VAD pre-filter and stricter decoding thresholds removed most of it.
After a scheduler outage, our daily orders DAG caught up two missed runs and both loaded the same day while another day was skipped. The task computed its date from the wall clock instead of the run's data interval.
A moving-average strategy showed strong backtest returns and nothing like it in paper trading. The backtest computed the signal from a candle's close and filled the order at that same close, a price nobody could trade at once the signal was known. Shifting signals to the next bar fixed the simulation.
We added cache_control to a 6,000-token system prompt and the bill went up instead of down. Every request wrote the cache and none read it, because the prompt prefix changed on every call. A stable prefix brought cache reads to 92 percent.
Every POST to our Django admin returned 403 after we moved TLS termination to a Traefik ingress. Django compared the browser's https Origin with a request it believed was plain http. Two settings fixed it without weakening CSRF.
Our RAG prototype was one folder where the PDF loader, the vector client and the API route imported each other freely. Here is the layout we moved to: four packages with enforced boundaries, a shared contract module, versioned indexes and an eval gate in CI.
A practical guide to time handling that survives DST switches: when to store UTC instants, when to keep local time plus an IANA zone, how gaps and overlaps behave, which database types to use, and how to test with fixed clocks.
Our monitoring service polled depth and klines for 40 symbols every two seconds. During a volatile hour it hit 429 Too much request weight, kept retrying, and earned an HTTP 418 IP ban that also blocked our trading bot on the same IP. Weight-aware limiting and websockets fixed it.
A handful of customers were charged twice for a single order, always around deploys. Kafka delivered the event more than once, which it is allowed to do. The real bug was a check-then-charge consumer with no atomic claim and no idempotency key sent to the payment provider.