Third-party APIs, webhooks, payment gateways, OAuth providers.
Two withdrawals hit the same wallet within milliseconds, both saw enough balance, and the user spent more than they had. The code ran inside a transaction, but READ COMMITTED does not protect a read-then-write. We compare atomic updates, row locks and stricter isolation.
A field guide to receiving webhooks reliably: verify signatures on the raw body, respond fast, queue the work, deduplicate by event ID, tolerate out-of-order delivery, replay safely and monitor the whole pipeline. With code you can adapt.
We launched local pricing in JPY, KRW and VND and a customer in Osaka was charged 398,000 yen for a 3,980 yen course. Our code multiplied every price by 100 before sending it to Stripe. How we found it, refunded it and built a per-currency exponent table with tests.
Our partner OAuth server matched redirect_uri by prefix, and our login page followed any next URL. Chained together with a legacy implicit flow, an access token could land on an outside domain. We moved to exact matching, relative-only next and code plus PKCE.
Sign in with Google worked locally and on staging but failed in production with redirect_uri_mismatch. Behind the TLS proxy our FastAPI app built an http:// callback because Uvicorn did not trust the proxy's headers. How we traced and fixed it.
A narrative of one on-call night on a subscription payments platform: the 01:40 page, a settlement job stuck on a lock, a provider returning timeouts that were actually successes, the decisions under pressure, and the blameless review that followed. Details changed, mistakes and lessons real.
After we tightened DMARC to quarantine, Gmail started filing our password resets and receipts as spam. Both SPF and DKIM passed, but for amazonses.com, not our domain. Here is how we read the headers, aligned both, and kept it that way.
A slow accounting API pushed our PayPal webhook handler past the nginx timeout. PayPal saw a 504 and retried, while the first request was still running, and 41 wallets were topped up twice. Postmortem plus the idempotent design we moved to.
We designed a multi-tenant payment platform for debt-collection agencies: card and ACH, one-time and recurring plans, settlement offers, disputes and reconciliation. Here is how we kept PCI scope small, why the ledger is double-entry and immutable, and how retries stay safe.
Our PDF invoices and the amounts charged through Stripe disagreed by a cent on 214 invoices in one month. Floats, toFixed and rounding at every step were to blame. We moved to integer minor units with one explicit rounding point.
A handful of customers were charged twice for a single order, always around deploys. Kafka delivered the event more than once, which it is allowed to do. The real bug was a check-then-charge consumer with no atomic claim and no idempotency key sent to the payment provider.
After a harmless-looking Express refactor, every Stripe webhook failed verification and paid bookings stayed pending. A global JSON parser consumed the body before constructEvent saw it. How we found it, fixed the route and replayed the missed events.
On November 3, 2024, our 01:30 billing run fired at 01:30 CDT and again at 01:30 CST, charging 1,284 subscriptions twice. The in-house scheduler matched local wall-clock time. We moved schedules to UTC and added idempotency keys, a run ledger and a unique constraint.