During an investigation we needed to know who changed a security group in one of 14 AWS accounts three months earlier, and the logs were gone. How we set up one organization trail into a protected log archive account, an Athena table with partition projection, and the queries we use most.
The incident response lifecycle turned into a runbook a small team can follow under pressure: preparation, detection and analysis, containment, eradication, recovery and the post-incident review, with roles, evidence handling and communication templates.
Our first RCA for a duplicate-invoice incident ended at 'engineer ran the job twice'. That was true and useless. Rerunning it with branching 5 whys, a fishbone and a causal chart found six conditions, three of which would have caused the next incident. Here is the method.
A narrative of one on-call night on a subscription payments platform: the 01:40 page, a settlement job stuck on a lock, a provider returning timeouts that were actually successes, the decisions under pressure, and the blameless review that followed. Details changed, mistakes and lessons real.
A practical guide to blameless postmortems: how to build a timeline, measure impact, find contributing factors instead of a culprit, write action items that get done, and run the review meeting so people tell the truth.