Most Kafka incidents we get paged for come down to five concepts: partitions, leaders, the in-sync replica set, producer acks and consumer group assignment. This is the mental model we use to read the symptoms and find the cause fast.
After a spot node reclaim, our Spring Cloud Gateway sent traffic to instances that no longer existed for over six minutes. Eureka had entered self-preservation and stopped evicting, while three layers of caches added their own delay. Here is how we cut stale routing to seconds.
Our monthly 'top 3 sellers per region' bonus report left out a sales rep who tied for third, and a rerun picked the other one. ROW_NUMBER breaks ties arbitrarily. We show how to pick RANK, DENSE_RANK or WITH TIES on purpose.
A security review found refresh tokens sitting in plain text in our Flutter app's preferences file. This guide sorts local data into three buckets, explains what shared_preferences, Hive (and its maintained hive_ce fork) and flutter_secure_storage actually do on each platform, and covers the uninstall and backup traps.
Instances in a freshly added private subnet timed out on dnf and never showed up in Session Manager, while identical instances in older subnets were fine. The subnet had silently fallen back to the VPC main route table, which had no NAT route.
Two withdrawals hit the same wallet within milliseconds, both saw enough balance, and the user spent more than they had. The code ran inside a transaction, but READ COMMITTED does not protect a read-then-write. We compare atomic updates, row locks and stricter isolation.
We moved an 800-million-row events table to monthly range partitions with a cutover lock of under a second. The trick was to attach the existing table as one big legacy partition, with every slow step done beforehand under weak locks.
The analytics team started consuming our order events and fulfilment began missing about half of all new orders. Nothing errored. Both services were reading the same RabbitMQ queue, and a queue hands each message to exactly one consumer. We fixed it with an exchange and one queue per service.
Our ECS service started, registered, failed health checks and got replaced, over and over. Two separate problems stacked: the container moved to port 8080 while the target group still probed 80, and a slow JVM start ran out the clock with no grace period.
Insert throughput on our events table fell by a factor of six as it passed 400 million rows, while CPU stayed idle. The cause was random UUIDv4 keys scattering writes across the whole primary key index. Postgres 18 uuidv7() brought it back, without changing the column type.
After a harmless-looking prompt change, our agent's time to first token went from 0.5 s to 1.6 s and GPU prefill load tripled. vLLM's prefix cache hit rate had dropped from 87% to 2%. The cause was one line at the top of the system prompt, plus tool schemas serialised in random order.
A stash pop hit a merge conflict, a reset cleared the mess, and a stash clear tidied up the list. Two days of uncommitted work were gone. Git had kept the stash commit as an unreachable object, and we got it back with git fsck. Here is how, and the habits that make it unnecessary.
A security review found that our password reset tokens and temporary passwords came from Python's random module. Its Mersenne Twister generator is fast and statistically good, but not unpredictable. We moved to secrets, hashed tokens at rest, and added Bandit to CI.
Our nightly pg_dump job reported success every night for four months. On the day we needed one, every dump in the bucket was truncated. A replica query cancellation plus a missing pipefail hid it. Here is the postmortem and the restore-tested backup job we run now.
Our first RCA for a duplicate-invoice incident ended at 'engineer ran the job twice'. That was true and useless. Rerunning it with branching 5 whys, a fishbone and a causal chart found six conditions, three of which would have caused the next incident. Here is the method.
An LRU cache is a hash map plus a linked list. A database index is a B-tree. A rate limiter is often a sorted set. When we started reading system components as data structures, capacity questions and odd latency curves became much easier to explain.
A traffic spike scaled our API from 8 to 30 pods, each with a 20-connection pool, and Postgres started refusing connections. Raising max_connections would have made it worse. We put PgBouncer in transaction mode in front and sized pools from the database side.
Celery workers stopped picking up thumbnail jobs one by one, with no errors in the logs. A Popen.wait() on a chatty child filled the 64 KiB pipe buffer and both processes waited on each other. We fixed the deadlock, added timeouts, and learned why timeout alone is not enough with shell=True.
Writing a 200-character regex in one go is how parsing bugs are born. We build patterns the way we build code: named parts, explicit groups, alternation that stays contained, lookaheads for extra rules, then extract every field from log lines and key=value strings with tests that keep it honest.
Our session-end events showed up for about a third of sessions on desktop and almost none on mobile. The tracker posted with fetch inside an unload handler that browsers cut short or never fire. Flushing on visibilitychange with sendBeacon brought delivery above 97 percent.
Connecting WhatsApp, Telegram or Slack to an LLM agent that can run tools on your server is a few hundred lines of glue and a large attack surface. This is the architecture we use: normalised channel adapters, sender pairing, a locked-down tool sandbox and hard budget caps.
Users editing several rows quickly lost some of their changes with no error at all. Our autosave saga used takeLatest, which cancelled the save for row A as soon as row B fired, leaving the UI saying Saved while state rolled back. Per-entity concurrency fixed it.
Field inspectors kept losing form submissions in basements and car parks. We rebuilt the app around a local-first write path: an IndexedDB outbox, an idempotent sync endpoint, version-based conflict detection and a UI that tells the truth about what is synced. Full working code included.
We swapped Docker for rootless Podman on a small RHEL host and the stack failed twice: rootlessport refused port 80, then PostgreSQL could not write its bind-mounted data directory. Both come from running without root: unprivileged ports, user namespaces and SELinux labels.
Kafka has no message priority, so a marketing blast on the same topic can delay password reset codes by twenty minutes. We compare the designs that actually work: separate topics with dedicated consumers, pausing bulk partitions in one consumer, and broker quotas.