One on-call night on a payments platform: a stuck settlement job, partial captures and the review

Postmortems usually read like reports: timeline, root cause, actions. They rarely describe what it is like to be the person holding the pager, with incomplete information and money moving. This is a narrative of one on-call night on the subscription payments platform I worked on, written up with the team's permission. Names, amounts and some details are changed; the sequence of mistakes and decisions is real.

The platform charges about 40,000 subscriptions on the first night of each month, in a batch that starts at 01:00 UTC: it authorises and captures payments through a payment provider, then a settlement job records the results in our ledger and marks invoices as paid.

01:40 - the page

The alert said: "Invoice settlement backlog above threshold: 18,000 unsettled". Normally the settlement job processes captures within minutes. I opened the dashboard on my phone, saw the backlog growing and the settlement job's throughput at zero, and got out of bed.

First lesson of the night, relearned: read the alert, then look at more than one graph before acting. The charge job's graph showed captures succeeding. The settlement job's graph was flat. So money was being taken, but our records were not catching up.

01:48 - what is the settlement job doing?

The job's logs ended at 01:21 with nothing after. The process was alive. A thread dump showed every worker thread waiting on a database lock. In PostgreSQL:

SELECT pid, now() - xact_start AS age, state, wait_event_type, left(query, 80)
FROM pg_stat_activity
WHERE datname = 'billing' AND state <> 'idle'
ORDER BY xact_start;

The oldest transaction, 27 minutes old, was idle in transaction, from a reconciliation script. It held a row lock on the ledger's monthly summary row, which every settlement transaction needed to update.

01:55 - the first decision

The obvious move was to kill the script's session. Before doing that, I checked what the script was: a finance reconciliation run started manually at 01:15 by a colleague in another time zone, who had opened a transaction in a database console, run a query, and gone to a meeting.

Killing it would roll back nothing important (it had only read and locked), so I terminated the session and posted in the incident channel what I had done and why:

SELECT pg_terminate_backend(48812);

Settlement throughput jumped. The backlog started to drain. I thought the incident was over. It was not.

02:20 - the second problem

As the backlog drained, a new alert fired: "Capture error rate above 5%". The charge job was now getting timeouts from the payment provider for a subset of captures, and its retry logic was retrying them. In the logs:

02:19:44 capture invoice=inv_88123 attempt=1 result=timeout after 30s
02:20:15 capture invoice=inv_88123 attempt=2 result=declined code=already_captured

already_captured. The first attempt had timed out from our side, but the provider had captured the payment. Our retry with a new request was correctly refused by the provider, but our code treated that response as a decline, and marked the invoice as failed. Customers whose payments had succeeded were being told their payment had failed, and dunning emails were queued.

02:30 - stop the bleeding

The priority shifted. Charges were succeeding at the provider, but our side was mislabelling a growing number of them. Options:

1. Stop the charge job: no new mislabels, but no new charges either.
2. Let it run and fix the records later.
3. Hot-fix the retry handling at 02:30 in the morning.

I chose the first, and paused the dunning email queue. In a payments system, an incorrect state is worse than a delay: a delayed charge can be retried in the morning; an incorrect "failed" status triggers emails, service suspensions and support tickets. I paged the payments team lead, because stopping the monthly charge run has business impact and was not a decision I wanted to make alone. She agreed within five minutes.

02:45 - understanding the timeouts

With the job paused, I looked at why the provider was timing out. Their status page showed nothing. Our latency graph to their API showed p99 rising from 800 ms to over 30 seconds starting at 02:12, right after the settlement backlog started draining. The settlement job, catching up, was calling the provider's API to fetch capture details for 18,000 invoices at once, on the same connection pool and the same rate limit as the charge job. We were throttling ourselves.

03:30 - recovery

The plan, written in the channel before executing:

1. Let settlement finish draining with reduced concurrency (from 32 to 8 workers).
2. Write a query listing invoices marked "failed" with a provider response of already_captured since 01:00.
3. For each, fetch the capture from the provider by our idempotency key and correct the invoice and ledger entries.
4. Cancel queued dunning emails for corrected invoices.
5. Resume the charge job with concurrency halved.

Step 3 relied on one good design decision made years earlier: every capture request carried an idempotency key derived from the invoice id and attempt group, so we could ask the provider "what happened to the capture with this key?" and get an authoritative answer.

By 04:50, 1,214 invoices had been corrected, no customer had received a wrong email (the queue was paused in time), and the charge run resumed. It finished at 06:10, five hours late.

The review

We held a blameless review two days later. The goal was not to find who did what wrong, but why the system let each step happen.

What went well

• The settlement backlog alert fired before customers noticed anything.
• Idempotency keys on captures made correcting records possible and safe.
• Pausing dunning early prevented customer-facing damage.
• Decisions and actions were written in the channel as they happened, which made the timeline easy to rebuild.

What went badly

• A manual console session could hold a lock on a critical row for 27 minutes.
• The retry logic retried an ambiguous timeout with a new request, instead of the same idempotency key, and treated already_captured as a failure.
• Settlement and charges shared a connection pool and rate limit with the provider.
• There was no runbook for "stop the monthly charge run safely".

Where we were lucky

• The finance colleague's script only read data.
• The dunning queue happened to batch emails every 30 minutes, so nothing had been sent before we paused it.

Actions

What I took away personally

• In money systems, prefer stopping to guessing.
• Write in the incident channel before acting, not after. It forces a moment of thought and gives others a chance to object.
• Wake up the person who owns the business decision; it is their call and their right.
• Idempotency is not a feature for the happy path. It is what makes recovery possible at 3 a.m.

References

• Google SRE Book, Postmortem Culture: Learning from Failure: https://sre.google/sre-book/postmortem-culture/
• PostgreSQL, idle_in_transaction_session_timeout: https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT
• PostgreSQL, pg_stat_activity: https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW
• IETF draft, The Idempotency-Key HTTP Header Field: https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/