Memory leaks, slow queries, caching, high-traffic incidents.
Users were logged out at random, mostly in the afternoon. Sessions and a fast-growing cache shared one Redis with allkeys-lru, so every hourly cache warm-up evicted idle sessions. Separate DB numbers would not have helped; separate instances did.
The same query ran in 2 ms from psql and 4 seconds from our Go service, but only for some tenants and only after a connection had been alive for a while. Postgres had switched the prepared statement to a generic plan on its sixth execution.
EXPLAIN ANALYZE prints a wall of nodes, costs and loops that most of us skim for the word Seq Scan. This guide walks a real slow query node by node: estimates vs actuals, loops, buffers, and the fixes each pattern points to.
Our flash sales pushed traffic to eight times normal in under a minute, and autoscaling arrived five minutes late. This guide covers what actually kept us up: the right balancing algorithm, pre-scaling around known events, queues for deferrable work and shedding load on purpose.
Every ten minutes our catalog API spiked to 4-second responses and the database hit 100% CPU. A hot Redis key expired and 600 requests rebuilt it at once. Single-flight, a short Redis lock, stale-while-revalidate and TTL jitter fixed it.
When MySQL sits at 100% CPU, the guilty query is rarely the slowest one; it is usually a fast query run millions of times. A practical sequence for finding it: the processlist for right now, performance_schema statement digests for the last hours, the sys schema for shortcuts, and pt-query-digest for slow log history.
Our monitoring service polled depth and klines for 40 symbols every two seconds. During a volatile hour it hit 429 Too much request weight, kept retrying, and earned an HTTP 418 IP ban that also blocked our trading bot on the same IP. Weight-aware limiting and websockets fixed it.
Our job board listing took 1.8 seconds and ran 312 queries per page. Blade was lazy loading company, tags and application counts for every row. We fixed it with with(), withCount() and made lazy loading throw in development.
Our thumbnail service hit Errno 24 at about 1,000 concurrent connections even though ulimit -n showed 65535. systemd services never read limits.conf. We set LimitNOFILE in a drop-in and fixed a client leak that was eating descriptors.
Our monthly-partitioned orders table was supposed to make the daily report touch one partition. EXPLAIN showed all 36. A date cast on the partition key and a join-driven filter were blocking pruning, and two small query rewrites brought it back.
During a launch our nginx reverse proxy returned errors and logged worker_connections are not enough, at well under the expected capacity. Every proxied request holds a client and an upstream connection, and the file descriptor limit was lower still. How we sized it properly.
During peak traffic a third of our traces arrived with missing spans, and the ones we needed for incident review were exactly the ones gone. The Collector was refusing data at its memory limit and dropping from a full export queue. Here is how we measured the loss and sized the pipeline.
Our text classification service crashed with CUDA out of memory every few hours, always on a big batch. Logits kept for drift monitoring were holding autograd graphs alive, batches were sized by count not tokens, and the allocator was fragmented. How we measured each one and fixed it.
Our storefront BFF pods restarted every few hours with OOMKilled and exit code 137, then fell into CrashLoopBackOff during the holiday peak. A max-old-space-size inherited from the VM days was double the container memory limit. Sizing the heap to the limit and bounding a cache fixed it.
Our price streaming API grew from 180 MB to 1.4 GB of heap in a day and was restarted by the orchestrator every night. A listener was attached to a shared EventEmitter on every request and never removed. Heap snapshots proved it; removing listeners on close fixed it.
A routine rolling deploy restarted one realtime pod holding 50,000 WebSocket connections. Every client reconnected within two seconds, the auth and history lookups melted, and the storm kept itself alive for 25 minutes. Here is how we drain sockets, pace reconnects and make resume cheap.
Our invoice list took 3 seconds even though a composite index covered both filter columns. The range column was first, and two other queries wrapped the column in a function or a cast. Reordering and rewriting fixed all three.
Our embedding API logged a tokenizers fork warning thousands of times a day and tokenisation got slower after we enabled Gunicorn preload. The tokenizer was used in the master process before workers forked. Loading it after fork fixed both.