A single 40-character form value froze an API worker for minutes. A nested quantifier in a validation regex made Python's backtracking engine try exponentially many paths. We show how we proved it, rewrote the pattern and added RE2 and CI checks.
A routine dependency update in our scoring service upgraded scikit-learn, and the logs filled with InconsistentVersionWarning when loading models pickled with an older version. One model's predictions shifted slightly. We pinned versions with each model, added load-time checks and moved to portable formats.
Our cleaning job filled missing countries and lowercased emails, yet the output table still had nulls and mixed case. Chained assignment wrote to a temporary copy, and the warning that said so had been silenced. We fixed it with .loc and turned on Copy-on-Write.
Our nightly export reported the right row count but wrote an empty CSV. A refactor had turned a list into a generator, and the first loop consumed it. Here is how we traced it, the three ways to fix it, and the type hints that now catch it.
Our expense app sends receipt photos to a vision model. The feature cost six times our estimate and p95 latency was 9 seconds. Phones upload 12-megapixel photos and we sent them as-is. Resizing and cropping before upload cut cost by 78 percent with no accuracy loss.
The analytics team started consuming our order events and fulfilment began missing about half of all new orders. Nothing errored. Both services were reading the same RabbitMQ queue, and a queue hands each message to exactly one consumer. We fixed it with an exchange and one queue per service.
A security review found that our password reset tokens and temporary passwords came from Python's random module. Its Mersenne Twister generator is fast and statistically good, but not unpredictable. We moved to secrets, hashed tokens at rest, and added Bandit to CI.
An LRU cache is a hash map plus a linked list. A database index is a B-tree. A rate limiter is often a sorted set. When we started reading system components as data structures, capacity questions and odd latency curves became much easier to explain.
A handful of tests in our Python service failed only when the whole suite ran, and only in some orders. A session-scoped fixture returned a mutable settings object that one test modified. How we proved it with random ordering and bisection, and how fixture scopes, factories and monkeypatch keep tests isolated.
Celery workers stopped picking up thumbnail jobs one by one, with no errors in the logs. A Popen.wait() on a chatty child filled the 64 KiB pipe buffer and both processes waited on each other. We fixed the deadlock, added timeouts, and learned why timeout alone is not enough with shell=True.
Writing a 200-character regex in one go is how parsing bugs are born. We build patterns the way we build code: named parts, explicit groups, alternation that stays contained, lookaheads for extra rules, then extract every field from log lines and key=value strings with tests that keep it honest.
Most agent demos are one big loop around a model. In production we replaced ours with four explicit patterns: a router, a planner with a fixed step budget, a bounded reflection pass and an approval gate for side effects. Here is how each works and where it fails.
We added a logging middleware to a FastAPI service and a server-sent events endpoint stopped streaming smoothly: events arrived in bursts, timings were wrong and disconnects went unnoticed. The middleware used BaseHTTPMiddleware. Why it interferes, and how pure ASGI middleware fixes it.
A practical first pass at regular expressions with Python's re module: the tokens you use every day, quantifiers, anchors, and the difference between search, match and fullmatch, ending with one pattern that safely validates an upload filename.
Our nightly product description job used the Batch API to save cost. One night the batch expired with 30 percent unfinished, and our importer joined results by line position, so descriptions landed on the wrong products. We now join by custom_id and handle every terminal status.
We launched local pricing in JPY, KRW and VND and a customer in Osaka was charged 398,000 yen for a 3,980 yen course. Our code multiplied every price by 100 before sending it to Stripe. How we found it, refunded it and built a per-currency exponent table with tests.
A customer saw a snippet from another company's document in an AI answer. A new 'related questions' feature queried the vector store through a helper that did not apply the tenant filter. We moved tenant scoping into the repository layer so no code path can skip it.
A new bot received order updates for one hour after start, then nothing, while the websocket stayed connected. The listenKey expires after 60 minutes without a keepalive. We fixed keepalives and expiry handling, then moved to the WebSocket API subscription before listenKey retirement.
A step-by-step build of a small Model Context Protocol server in Python with tools, resources and prompts, then how we test it three ways: the MCP Inspector, a scripted client in pytest and a real AI client, before anyone on the team relies on it.
Our support assistant quoted an old refund policy that had been removed from the help center weeks earlier. The ingestion pipeline only ever upserted chunks and never deleted them. We added source-of-truth sync, document versions and a nightly reconciliation job.
A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.
Our Ray preprocessing job for image embeddings failed after an hour with OutOfMemoryError. Ray schedules tasks by CPU count, so it packed 16 tasks that each needed 3 GB onto nodes with 30 GB. Declaring memory per task, smaller blocks and a right-sized object store fixed it.
Prompt and response logs are the fastest way to build an accidental PII database. This guide shows where personal data enters an LLM feature, how to redact it before logging and before the provider, and how to keep retention and access under control.
Our daily signals disagreed with the exchange chart, and intraday signals appeared and disappeared within the same bar. We resampled UTC millisecond timestamps in local time and included the still-forming last candle. UTC everywhere and using only closed candles fixed both problems.
A hedging bot meant to short perpetual futures placed spot sell orders instead, selling inventory it was supposed to hedge. In ccxt's unified symbols, BTC/USDT is spot and BTC/USDT:USDT is the linear perpetual. Explicit market types and assertions on market metadata fixed it.