A DuckDB report job queried Parquet files on S3 fine on a laptop and failed with HTTP 403 in its ECS task. Locally DuckDB used a persistent secret from personal credentials; in the container nothing told it to use the task role. How DuckDB secrets and the credential chain work, plus the KMS detail.
After a rolling restart of our Kafka cluster, the warehouse system was missing about 300 order events the producer had logged as sent. The producer used acks=1, so the leader acknowledged before followers had the data. How we proved it, and the settings that make acknowledged writes durable.
When a small self-hosted service lost its disk, we restored the MySQL Docker volume from last night's tar backup, and MySQL refused to start with InnoDB corruption errors. The backup had been a file copy of a running database. How we recovered from an older dump, and the backup approach and restore drills we use now.
Our PostgreSQL primary went read-only at 4 a.m. when its disk filled. The pg_wal directory held 380 GB, because a logical replication slot created for a CDC trial weeks earlier was inactive and still retained WAL. How we found the slot, recovered safely, and capped WAL retention with max_slot_wal_keep_size.
We moved a service on EKS from a database password to RDS IAM authentication. It worked at startup, then new connections failed with PAM authentication failed 15 minutes later: the app generated one token at boot and used it as the pool's password. How we generate tokens per connection.
Our game released 100 limited items at 20:00, and about 20,000 players tapped claim within seconds. 140 players received one. The endpoint checked remaining stock, then decremented it separately. How we proved the race and fixed it with an atomic conditional update, a constraint and a Redis gate.
When MySQL sits at 100% CPU, the guilty query is rarely the slowest one; it is usually a fast query run millions of times. A practical sequence for finding it: the processlist for right now, performance_schema statement digests for the last hours, the sys schema for shortcuts, and pt-query-digest for slow log history.