A narrative of one on-call night on a subscription payments platform: the 01:40 page, a settlement job stuck on a lock, a provider returning timeouts that were actually successes, the decisions under pressure, and the blameless review that followed. Details changed, mistakes and lessons real.
We designed a multi-tenant payment platform for debt-collection agencies: card and ACH, one-time and recurring plans, settlement offers, disputes and reconciliation. Here is how we kept PCI scope small, why the ledger is double-entry and immutable, and how retries stay safe.
Our PayPal checkout button rendered in development and staging but was an empty box in production. A new Content-Security-Policy set in Nginx blocked the PayPal SDK script, its iframes and its API calls. We fixed it per directive, with a nonce and a report-only rollout.
A handful of customers were charged twice for a single order, always around deploys. Kafka delivered the event more than once, which it is allowed to do. The real bug was a check-then-charge consumer with no atomic claim and no idempotency key sent to the payment provider.