Redis rate limiter let 3x the limit through: read-then-write race across pods

A partner shipping API suspended our account for twenty minutes on a Tuesday afternoon. Their contract allows 120 requests per second per customer, and our gateway is supposed to keep each of our tenants under 100 per second toward them. Their email included a chart: one of our tenants had peaked at 310 requests per second for about 40 seconds.

Our own dashboards disagreed. The limiter metrics showed the tenant capped at exactly 100 allowed per second, with the rest rejected. The partner's numbers and ours could not both be right, and the partner's were the ones that cost us money.

HTTP/1.1 429 Too Many Requests
Retry-After: 1200
X-Partner-Error: rate_limit_exceeded; account suspended for 1200s

Reproducing it

We pointed a load generator at staging with four API pods behind the load balancer, one tenant, and a 100 per second limit:

k6 run -e RATE=400 -e TENANT=t_load scripts/burst.js

At 400 requests per second offered, the downstream mock recorded between 180 and 290 accepted requests per second, varying run to run. With a single API pod the number was close to 100. The leak scaled with concurrency.

What we ruled out

• Clock differences between pods. The limiter keyed windows on Math.floor(Date.now() / 1000). Skew of a few milliseconds shifts window edges but cannot triple throughput.
• Requests bypassing the limiter. Access logs showed every request to the partner passed through the limiter middleware.
• Redis eviction. INFO stats showed evicted_keys:0, and the instance used noeviction.

Root cause

The limiter did its decision in application code between two Redis round trips:

// excerpt from the old middleware
const used = Number(await redis.get(key)) || 0;
if (used >= limit) return reject(res);
await redis.multi().incr(key).expire(key, 2).exec();
return next();

Each pod handles hundreds of concurrent requests on one event loop. Fifty requests could all GET the counter while it read 60, all decide there was room, and all increment it to 110. Our metrics counted the decision, not the resulting counter, so they reported 100 allowed while 300 went out. The MULTI around INCR and EXPIRE made those two commands atomic, but the GET and the decision were outside the transaction.

What is happening under the hood

Redis executes each command atomically, and a single client sees its own commands in order. Across clients there is no ordering guarantee between one client's GET and another client's INCR. Any check-then-act sequence spread over two round trips is a classic time-of-check to time-of-use race. More pods and more concurrency per pod widen the window.

MULTI/EXEC does not help with conditional logic, because commands are queued and you cannot branch on a value read inside the transaction. WATCH turns it into optimistic locking, but under a burst most transactions abort and retry, which adds load at exactly the wrong moment. A Lua script runs entirely on the Redis server while no other command executes, so read, decide and write become one step.

Fixed windows have a second, smaller issue: a client can send the full limit at the end of one second and again at the start of the next. A token bucket smooths that by refilling continuously and allowing a bounded burst.

Environment

Minimal reproduction

// race.ts - npx tsx race.ts
import Redis from "ioredis";
const redis = new Redis();
const key = "rl:t1:" + Math.floor(Date.now() / 1000);
let allowed = 0;

async function hit() {
  const used = Number(await redis.get(key)) || 0;
  if (used >= 100) return;
  await redis.multi().incr(key).expire(key, 2).exec();
  allowed++;
}

await Promise.all(Array.from({ length: 500 }, hit));
console.log({ allowed, counter: await redis.get(key) });
redis.disconnect();

On a laptop this prints something like { allowed: 500, counter: '500' }, because all 500 GETs are pipelined before the first INCR lands.

Who is affected

Any limiter where the decision happens in the client after reading shared state: GET then SET, ZCARD then ZADD, or reading a hash and writing it back. Pure INCR plus a comparison on the returned value is safe for fixed windows, because INCR returns the post-increment value atomically. In-memory per-pod limiters are a separate bug: they allow limit times pod count.

The fix

We replaced the middleware logic with a token bucket implemented as a single Lua script. Redis server time is used so pods do not need synchronised clocks.

-- token_bucket.lua
-- KEYS[1] bucket key
-- ARGV[1] capacity, ARGV[2] refill per second, ARGV[3] tokens requested
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])
local want     = tonumber(ARGV[3])

local t   = redis.call('TIME')
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

local state  = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(state[1])
local ts     = tonumber(state[2])
if tokens == nil then
  tokens = capacity
  ts = now
end

local elapsed = math.max(0, now - ts)
tokens = math.min(capacity, tokens + elapsed * rate / 1000)

local allowed = 0
local retry_ms = 0
if tokens >= want then
  tokens = tokens - want
  allowed = 1
else
  retry_ms = math.ceil((want - tokens) * 1000 / rate)
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity * 1000 / rate) + 1000)
return { allowed, retry_ms }

// limiter.ts
import Redis from "ioredis";
import { readFileSync } from "node:fs";

const redis = new Redis(process.env.REDIS_URL!);
redis.defineCommand("tokenBucket", {
  numberOfKeys: 1,
  lua: readFileSync(new URL("./token_bucket.lua", import.meta.url), "utf8"),
});

export async function take(tenant: string, capacity = 100, rate = 100) {
  const [allowed, retryMs] = await (redis as any).tokenBucket(
    `rl:partner:{${tenant}}`, capacity, rate, 1,
  );
  return { allowed: allowed === 1, retryMs: Number(retryMs) };
}

// in the Fastify hook
const { allowed, retryMs } = await take(req.tenantId);
if (!allowed) {
  reply.header("Retry-After", Math.ceil(retryMs / 1000));
  return reply.code(429).send({ error: "rate_limited" });
}

defineCommand loads the script with EVALSHA and falls back to EVAL if the script cache was flushed, so a Redis restart does not break the limiter. The braces in the key keep every key for a tenant on one hash slot if we move to Redis Cluster.

Deciding what happens when Redis is down

The old code failed open silently: a Redis timeout threw, the catch block called next(). For this limiter, failing open is how we get suspended by the partner, so we fail closed for partner-bound calls and return 503 with a short Retry-After. For our own read endpoints the same failure fails open with a local per-pod fallback limit of limit / podCount. The choice depends on what the limit protects; it should be explicit, not a side effect of a catch block.

Alternatives and trade-offs

• Fixed window with INCR only. Call INCR, set EXPIRE when the result is 1, compare the returned value. Atomic and cheap; allows up to 2x the limit at window edges.
• Sliding window log in a sorted set. Precise, but stores one member per request, which is expensive at high limits.
• Gateway-native limiting. Envoy's global rate limit service or an API gateway product does the same work outside the app. We kept it in the app because limits depend on tenant plan data.

How to test the fix

npx vitest run test/limiter.concurrency.test.ts
k6 run -e RATE=400 -e TENANT=t_load scripts/burst.js

The concurrency test fires 1,000 parallel take() calls against a fresh bucket of capacity 100 and asserts exactly 100 are allowed. The k6 run against staging now shows the downstream mock at 99 to 101 requests per second across 4 and 12 pods.

Confirming in production

We added a counter on the partner client itself, after the limiter, so we measure what actually leaves the building. A week after the release, the per-tenant peak in that series was 100.4 requests per second (sampling noise at second boundaries), and the partner reported no limit breaches. Redis CPU rose from 11 percent to 14 percent at peak; the script runs in roughly 20 microseconds.

redis-cli --latency-history -i 5
redis-cli SLOWLOG GET 10

Neither showed the script in the slow log at the default 10 ms threshold.

Regression guard

• The concurrency test runs in CI against a real Redis container, not a mock.
• Metrics for "allowed by limiter" and "sent to partner" sit on the same panel with an alert if the second exceeds the first by more than 2 percent.
• Code review rule: limiter logic lives only in Lua scripts in the limits/ package.

Security implications

A leaky limiter is a denial-of-service and cost risk: abusive clients get more capacity than their plan, and a third party can suspend all tenants because of one. The fix narrows that. Two things to keep in mind: the bucket key must use an authenticated tenant id, never a client-supplied header, or callers can rotate keys to dodge limits; and failing open on Redis errors would reopen the hole during an outage, so the failure mode is now chosen per route.

Prevention checklist

• Never split read, decide and write across round trips for shared counters.
• Count what reaches the protected resource, not what the limiter decided.
• Use server time inside the script rather than each pod's clock.
• Choose fail-open or fail-closed per route and test it by killing Redis in staging.

References

• Redis docs, Scripting with Lua: https://redis.io/docs/latest/develop/interact/programmability/eval-intro/
• Redis docs, TIME command: https://redis.io/docs/latest/commands/time/
• Redis docs, Transactions (MULTI, EXEC, WATCH): https://redis.io/docs/latest/develop/interact/transactions/
• ioredis README, Lua scripting and defineCommand: https://github.com/redis/ioredis
• RFC 6585, section 4 (429 Too Many Requests): https://www.rfc-editor.org/rfc/rfc6585