Automated trading fails fast: a bug can place hundreds of orders before a human notices. This guide covers the monitoring and stopping mechanisms we run for every bot: heartbeats, position and loss limits, cancel-all on disconnect, a tested kill switch, and runbooks for the first five minutes.
Our order worker stopped completely for 47 minutes and the throughput alarm never fired. With no datapoints arriving, the alarm moved to INSUFFICIENT_DATA instead of ALARM, and only ALARM had an action. The fix was choosing missing data treatment per alarm.
Our nightly invoice export stopped running and nobody noticed for nine days. One Job had hung on a network call with no timeout, and with concurrencyPolicy Forbid every later schedule was skipped. Deadlines on the Job, timeouts in code and an alert on last success fixed it.
Our PostgreSQL primary went read-only at 4 a.m. when its disk filled. The pg_wal directory held 380 GB, because a logical replication slot created for a CDC trial weeks earlier was inactive and still retained WAL. How we found the slot, recovered safely, and capped WAL retention with max_slot_wal_keep_size.
After we changed the scrape interval from 15 to 60 seconds to save money, our request-rate panels went blank in places and alerts flapped. rate() needs at least two samples in its range, and our queries used [1m]. Why the range must be several scrape intervals, how $__rate_interval works, and how we fixed alerts too.
Our customer portal went dark at 07:12 when its Let's Encrypt certificate expired. Certbot's timer had run twice a day for a month and failed every time, because a firewall change blocked port 80. Here is how we traced it, fixed it and made expiry impossible to miss.