Postgres primary disk filled overnight: an abandoned CDC replication slot kept every WAL file

At 04:12 the on-call engineer was paged: the orders database was rejecting writes. The PostgreSQL log:

2026-01-23 04:11:58 UTC [1203] PANIC:  could not write to file "pg_wal/xlogtemp.1203": No space left on device
2026-01-23 04:11:58 UTC [1] LOG:  WAL writer process (PID 1203) was terminated by signal 6: Aborted
2026-01-23 04:11:58 UTC [1] LOG:  terminating any other active server processes

PostgreSQL had crashed and restarted in a loop, unable to write WAL. The data volume was 500 GB; normal usage was about 110 GB.

Where the space went

du -sh /var/lib/postgresql/16/main/base /var/lib/postgresql/16/main/pg_wal

104G    /var/lib/postgresql/16/main/base
382G    /var/lib/postgresql/16/main/pg_wal

The tables were their normal size. The write-ahead log directory, which usually held a few gigabytes, had 382 GB.

Why was WAL not removed?

PostgreSQL removes old WAL segments after checkpoints, unless something still needs them: a replica that has not received them, wal_keep_size, an archiving failure, or a replication slot. After freeing a little space by moving old log files off the volume, we started the server and checked:

SELECT slot_name, slot_type, plugin, active, restart_lsn, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots;

   slot_name    | slot_type |  plugin  | active | restart_lsn  | wal_status | retained
----------------+-----------+----------+--------+--------------+------------+----------
 replica_1      | physical  |          | t      | 5A/1F2C3D40  | reserved   | 48 MB
 debezium_trial | logical   | pgoutput | f      | 4C/9A001288  | extended   | 379 GB

A logical replication slot named debezium_trial, inactive, retaining 379 GB of WAL.

Whose slot?

The data team had evaluated a change data capture (CDC) pipeline with Debezium six weeks earlier, in production, against a copy of one table. The trial ended, the connector was deleted, but deleting a Debezium connector does not drop its replication slot. The slot stayed, remembering the position where its consumer last confirmed, and PostgreSQL kept every WAL segment since then for a consumer that would never return.

What we ruled out

• WAL archiving failures. pg_stat_archiver showed archiving succeeding, with no failed count.
• A lagging physical replica. replica_1 was active and only 48 MB behind.
• A long-running transaction. That bloats tables, not WAL retention in this way.

Root cause

An inactive logical replication slot prevented PostgreSQL from recycling WAL. Slots exist to guarantee that a consumer can resume without losing changes, so PostgreSQL retains all WAL from the slot's restart_lsn indefinitely, by default. With no consumer and no retention limit, WAL grew with every write for six weeks until the disk was full.

What replication slots guarantee

A replication slot records how far a consumer has confirmed receiving changes. PostgreSQL keeps WAL (and, for logical slots, enough catalog information) from that point on, so a disconnected consumer can reconnect later and continue exactly where it stopped. Without a slot, WAL could be recycled while a consumer is away, and the consumer would need a full resynchronisation.

That guarantee is also the risk: a slot whose consumer is gone forever holds WAL forever.

Logical slots also hold back catalog cleanup

Logical slots keep a catalog_xmin, which prevents vacuum from removing old rows in system catalogs. On busy databases with frequent schema changes, catalog bloat is a second, slower symptom of an abandoned logical slot.

max_slot_wal_keep_size

Since PostgreSQL 13, max_slot_wal_keep_size limits how much WAL slots may retain. When a slot exceeds it, PostgreSQL stops retaining WAL for that slot, the slot's wal_status becomes lost, and the consumer must be resynchronised. The default, -1, means unlimited.

Who is affected

Any PostgreSQL server with replication slots: physical replicas configured with slots, logical replication subscriptions, and CDC tools such as Debezium or cloud migration services. Trials and decommissioned consumers are the typical source of abandoned slots. Managed services (RDS, Cloud SQL, Azure) behave the same, with the disk-full symptom often appearing as storage autoscaling bills first.

Environment

Recover: drop the abandoned slot

After confirming with the data team that nothing would ever use the slot again:

SELECT pg_drop_replication_slot('debezium_trial');
CHECKPOINT;

The CHECKPOINT lets PostgreSQL remove the WAL segments no longer needed. Within a minute, pg_wal was back to 3 GB, and writes resumed normally. We checked the physical replica's status to make sure it had not fallen behind during the outage.

Dropping a slot that is still in use breaks its consumer, which must then be resynchronised. Always confirm ownership first.

Prevent: cap WAL retained by slots

ALTER SYSTEM SET max_slot_wal_keep_size = '64GB';
SELECT pg_reload_conf();

64 GB is about a week of WAL at our rate, enough for a CDC connector or replica to recover from a long outage, and well below the free disk space. A slot that exceeds it is invalidated instead of filling the disk.

Monitor slots

An alert on retained WAL per slot and on inactive slots:

SELECT slot_name, active,
       pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes,
       wal_status
FROM pg_replication_slots;

Exported as metrics, with alerts:

• any slot inactive for more than one hour;
• retained WAL per slot above 10 GB;
• wal_status = unreserved or lost.

Ownership for slots

Every slot now has an owner recorded in our infrastructure inventory, and slots are created only through reviewed configuration. Debezium's connector configuration sets slot.drop.on.stop only for test environments; in production, the decommission runbook for a connector includes dropping its slot.

Separate WAL from data

We also moved pg_wal to its own volume. A future WAL explosion can still stop writes, but it can no longer fill the volume holding table data, and recovery is simpler.

Alternatives and trade-offs

• No limit, with monitoring only. Guarantees consumers never lose position, but relies on someone responding before the disk fills.
• Smaller limits. Protect the disk more, but force more frequent CDC resynchronisations after outages.

How to test the fix

On a staging server, create a slot without a consumer and generate WAL:

SELECT pg_create_logical_replication_slot('test_slot', 'pgoutput');
-- generate WAL, for example with pgbench, until retained exceeds the limit
SELECT slot_name, wal_status FROM pg_replication_slots WHERE slot_name = 'test_slot';
-- wal_status becomes 'lost' once retained WAL exceeds max_slot_wal_keep_size

Confirming in production

pg_wal stayed between 2 and 6 GB over the following weeks. The new alerts fired once, for a slot of a legitimate CDC pipeline that had stopped during a deploy; the data team restarted the connector within the hour, long before the limit.

Regression guard

• Slot metrics and alerts as above.
• max_slot_wal_keep_size set by configuration management on all PostgreSQL servers.
• Decommission runbooks for anything that creates slots include dropping them.

Security implications

None directly, but an abandoned slot is also a data exposure risk: anyone with replication privileges could connect to it and read every change since it was created, including personal data. Dropping unused slots and restricting the REPLICATION role attribute reduce that exposure.

Prevention checklist

• Treat every replication slot as a resource with an owner.
• Set max_slot_wal_keep_size so slots cannot fill the disk.
• Alert on inactive slots and on retained WAL per slot.
• Drop slots when decommissioning replicas and CDC connectors.
• Keep WAL on a separate volume from table data.

References

• PostgreSQL, Replication slots: https://www.postgresql.org/docs/16/warm-standby.html#STREAMING-REPLICATION-SLOTS
• PostgreSQL, max_slot_wal_keep_size: https://www.postgresql.org/docs/16/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE
• PostgreSQL, pg_replication_slots view: https://www.postgresql.org/docs/16/view-pg-replication-slots.html
• Debezium PostgreSQL connector, replication slot properties: https://debezium.io/documentation/reference/stable/connectors/postgresql.html