Redis distributed lock expired mid-job: two workers held it, fencing tokens fixed it

On the first business day of June our support queue filled up with the same complaint: monthly statements listed some charges twice. 214 of roughly 41,000 accounts were affected. Totals on those statements were wrong, and the PDF had been emailed before anyone noticed. Nothing had been deployed to the statement service for three weeks.

The statement job runs on a pool of six workers. Each worker picks an account, takes a Redis lock named after it, builds the statement lines, writes them to PostgreSQL, renders a PDF and releases the lock. The worker logs for one affected account told the story in two lines:

2025-06-02T00:41:07.112Z worker-3 lock acquired key=stmt:acct:88213 ttl_ms=30000
2025-06-02T00:41:39.904Z worker-5 lock acquired key=stmt:acct:88213 ttl_ms=30000
2025-06-02T00:41:52.310Z worker-3 inserted 37 statement_lines acct=88213
2025-06-02T00:41:52.388Z worker-3 lock released key=stmt:acct:88213
2025-06-02T00:41:55.026Z worker-5 inserted 37 statement_lines acct=88213

Two workers "held" the same lock 13 seconds apart, and the first one released a lock that by then belonged to the second.

Reproducing it

We reproduced it locally by adding a time.sleep(35) between acquiring the lock and writing rows, then starting two workers against the same account. It failed every time. In production the pause was not a sleep: the slow runs all lined up with a stalled call to the PDF rendering service, which had a 60 second client timeout, longer than our 30 second lock TTL.

What we ruled out

• Redis failover. A replica promotion can lose a recently written lock key. We checked INFO replication history and Sentinel logs: no failover that night.
• Clock skew. TTLs in Redis are measured by the Redis server, not by clients, so skew between worker hosts does not change when a key expires.
• Duplicate job scheduling. The scheduler enqueued each account exactly once; the second worker picked the account up because the retry logic saw "lock free" and treated the account as unclaimed.

Root cause

The lock was a lease, and our code treated it as ownership. SET key value NX PX 30000 guarantees that only one client holds the key at a time, but it says nothing about how long the client keeps running. When worker-3 stalled past 30 seconds, Redis expired the key, worker-5 legitimately acquired it, and worker-3 woke up and kept writing as if nothing had happened. A second bug made it worse: our release used a plain DEL, so worker-3 deleted worker-5's lock on the way out, which would have let a third worker in if one had been waiting.

What is happening under the hood

A Redis lock built on SET NX PX has three moving parts: the key, its expiry, and the value. The expiry exists so that a crashed holder cannot block everyone forever. That same expiry means the lock can disappear while the holder is still alive and still working, for reasons the holder cannot see: a stop-the-world GC pause, a slow network call, CPU starvation on a noisy host, or a process paused by the scheduler.

No amount of tuning removes this window. A longer TTL makes it less likely and makes crash recovery slower. A renewal thread helps, but a paused process cannot renew. The only robust answer is to let the resource being protected reject writes from a holder whose lease is stale. That is what a fencing token does: every successful acquisition gets a strictly increasing number, and the storage layer refuses any write that carries a number lower than the highest it has already seen.

Environment

Minimal reproduction

# broken_lock.py - run two copies at once: python broken_lock.py A & python broken_lock.py B
import sys, time, redis, psycopg

r = redis.Redis()
worker = sys.argv[1]
key = "stmt:acct:42"

while not r.set(key, worker, nx=True, px=3000):
    time.sleep(0.2)
print(worker, "acquired")

if worker == "A":
    time.sleep(5)  # stands in for the stalled PDF call; longer than the 3 s TTL

with psycopg.connect("dbname=demo") as conn:
    conn.execute("INSERT INTO statement_lines (account_id, worker) VALUES (42, %s)", (worker,))

r.delete(key)  # unsafe: may delete someone else's lock
print(worker, "released")

Run it with A starting first. B acquires at about three seconds, both insert, and A's DEL removes B's key.

Who is affected

Any design where a Redis (or etcd, ZooKeeper, Consul) lease protects a side effect in another system and the holder can run longer than the lease. Redlock across several Redis primaries does not change this; it addresses the availability of the lock service, not a paused client. Jobs that only use the lock as an optimisation (avoid doing the same cache warm-up twice) can tolerate it. Jobs where a duplicate write costs money or correctness cannot.