A canary that a human watches on a dashboard is only as good as that human's attention. This guide covers automated canary analysis: which metrics to gate on, how to size steps and durations, how to handle low traffic, and the traps that make canaries pass when they should fail.
Automated trading fails fast: a bug can place hundreds of orders before a human notices. This guide covers the monitoring and stopping mechanisms we run for every bot: heartbeats, position and loss limits, cancel-all on disconnect, a tested kill switch, and runbooks for the first five minutes.
Request-latency SLOs do not fit a pipeline of queues, functions and databases where users care about whether an order confirmation arrives in time. How we defined SLIs per hop, an end-to-end freshness SLO, how availability combines across steps, and how we alert on error budget burn.
Our first RCA for a duplicate-invoice incident ended at 'engineer ran the job twice'. That was true and useless. Rerunning it with branching 5 whys, a fishbone and a causal chart found six conditions, three of which would have caused the next incident. Here is the method.
A practical guide to blameless postmortems: how to build a timeline, measure impact, find contributing factors instead of a culprit, write action items that get done, and run the review meeting so people tell the truth.
A queue worker with Restart=always was found dead in the morning with result start-limit-hit. systemd had stopped restarting it after five crashes in ten seconds while Redis was briefly down. How to read the failure, tune StartLimit and RestartSec, and stop flapping services from staying dead.