After we raised a Lambda timeout to handle large PDFs, customers started getting the same invoice email twice. The SQS queue's visibility timeout was still 60 seconds, so slow messages reappeared and a second invocation picked them up mid-processing.
A field guide to receiving webhooks reliably: verify signatures on the raw body, respond fast, queue the work, deduplicate by event ID, tolerate out-of-order delivery, replay safely and monitor the whole pipeline. With code you can adapt.
A narrative of one on-call night on a subscription payments platform: the 01:40 page, a settlement job stuck on a lock, a provider returning timeouts that were actually successes, the decisions under pressure, and the blameless review that followed. Details changed, mistakes and lessons real.
During a volatile minute, our place-order request timed out and the bot retried. Both requests reached the exchange, and we held twice the intended position. A timeout does not mean the order failed. Client order ids, query-before-retry and an unknown state fixed it.
Our base playbook reported changed on every host on every run, so nobody trusted the drift report, and a duplicated sysctl line went unnoticed for months. The cause was shell and command tasks that Ansible cannot reason about; modules, creates and changed_when fixed it.
A slow accounting API pushed our PayPal webhook handler past the nginx timeout. PayPal saw a 504 and retried, while the first request was still running, and 41 wallets were topped up twice. Postmortem plus the idempotent design we moved to.
A handful of customers were charged twice for a single order, always around deploys. Kafka delivered the event more than once, which it is allowed to do. The real bug was a check-then-charge consumer with no atomic claim and no idempotency key sent to the payment provider.
On November 3, 2024, our 01:30 billing run fired at 01:30 CDT and again at 01:30 CST, charging 1,284 subscriptions twice. The in-house scheduler matched local wall-clock time. We moved schedules to UTC and added idempotency keys, a run ledger and a unique constraint.
Our onboarding page created two draft workspaces per visit in development. StrictMode was blamed and nearly disabled, but the real bug was a non-idempotent POST inside an effect with no cleanup, and it was also happening in production.