Retries done right: which errors to retry, retry budgets, and not stacking retries across hops

Every client library has a retry option, and most services turn it on. Retries are one of the cheapest ways to hide brief failures: a dropped connection, a pod restarting, a leader election. They are also one of the most common ways a small problem becomes an outage, because they add load exactly when a system is least able to handle it.

This post collects the rules our platform team applies when reviewing retry settings across about forty services. None of them is new; the value is in applying all of them together.

Rule 1: retry only what can succeed on a second attempt

Ask two questions about every failure: is it likely to be temporary, and is it safe to repeat the request?

Errors that are usually temporary

Client errors (4xx, except 408 and 429) mean the request itself is wrong. Retrying them only adds load.

Safe to repeat

HTTP defines GET, HEAD, OPTIONS, PUT and DELETE as idempotent: repeating them has the same effect as sending them once. POST and PATCH are not, unless the server supports an idempotency key.

This matters most for the ambiguous failures: a timeout or a 502/504 after the request was sent. The server may have done the work. Retrying a non-idempotent POST /payments after a timeout can charge twice. Our rule:

• Idempotent method or idempotency key present: retry ambiguous failures.
• Otherwise: retry only failures where the request certainly did not reach the server (connection refused, connection reset before any bytes were sent, 503 from a load balancer with no healthy targets).

Rule 2: back off exponentially, with jitter

Retrying immediately is the worst option: if the server is overloaded, an instant retry arrives while it is still overloaded. Exponential backoff waits longer after each failure. But plain exponential backoff has a problem when many clients fail at the same moment: they all retry at the same moments too, in synchronised waves.

Adding randomness, jitter, spreads them out. The "full jitter" approach picks a random delay between zero and the exponential cap:

import random

def backoff_delay(attempt: int, base: float = 0.1, cap: float = 5.0) -> float:
    """attempt starts at 1; returns seconds to wait before this attempt."""
    return random.uniform(0, min(cap, base * 2 ** attempt))

For attempts 1, 2, 3 that gives random delays up to 0.2 s, 0.4 s, 0.8 s, capped at 5 seconds. AWS's Architecture Blog compared jitter strategies and found full jitter reduced total work and completion time under contention compared with no jitter.

Also cap the number of attempts. For calls within a request path, we allow at most 2 retries (3 attempts total). For background jobs, more is fine, with longer caps.

Rule 3: a retry budget per client

Backoff limits how fast one request retries. It does not limit how much extra load all requests together generate. If a dependency fails completely, every request makes three attempts, and the dependency sees three times its normal traffic at the worst moment.

A retry budget caps retries as a fraction of normal traffic. For example: retries may add at most 10 percent on top of first attempts, measured over the last 10 seconds. When the budget is exhausted, failures are returned immediately without retrying.

class RetryBudget:
    """Allow retries up to `ratio` of recent first attempts, plus a small floor."""

    def __init__(self, ratio: float = 0.1, min_per_sec: float = 10, window: float = 10.0):
        self.ratio, self.min_per_sec, self.window = ratio, min_per_sec, window
        self.requests: deque[float] = deque()
        self.retries: deque[float] = deque()

    def _trim(self, q: deque, now: float) -> None:
        while q and q[0] < now - self.window:
            q.popleft()

    def record_request(self) -> None:
        self.requests.append(time.monotonic())

    def can_retry(self) -> bool:
        now = time.monotonic()
        self._trim(self.requests, now)
        self._trim(self.retries, now)
        allowed = self.ratio * len(self.requests) + self.min_per_sec * self.window
        if len(self.retries) < allowed:
            self.retries.append(now)
            return True
        return False

The floor (min_per_sec) lets low-traffic clients still retry occasional failures. Service meshes and some RPC frameworks have this built in: Linkerd's retry budgets, and gRPC's retry throttling, which uses a token bucket that failed calls drain and successful calls refill.

With a budget, a complete outage of a dependency adds about 10 percent load instead of 200 percent.

Rule 4: retry at one layer only

This is the rule most often broken, because each team configures retries for its own calls and nobody looks at the whole chain.

Consider a request that passes through three services, each retrying its downstream call 3 times (1 attempt plus 2 retries):

gateway (3 attempts) -> orders (3 attempts) -> inventory (3 attempts) -> database

If the database is down, the gateway's 3 attempts each cause orders to make 3 attempts, each of which causes inventory to make 3 attempts. Inventory receives 3 × 3 = 9 requests, and the database 27, for one user click. Retries multiply across layers.

Our convention:

• Retry at the layer closest to the failure that has the context to decide. Usually the immediate caller of the failing dependency.
• Every other layer does not retry that call, or retries only connection-level failures that never reached the next hop.
• Signal "do not retry" upstream. When a service has already retried and given up, it returns a response that tells callers not to retry (we use 503 without Retry-After plus an X-Retry-Exhausted: true header that our client library honours).

Rule 5: retries must fit the deadline

A retry is pointless if the caller has already given up. Each attempt and backoff must fit within the overall request deadline. If 200 ms remain and the next backoff is 400 ms, do not retry; fail now.

def call_with_retries(fn, deadline: float, budget: RetryBudget, max_attempts: int = 3):
    budget.record_request()
    for attempt in range(1, max_attempts + 1):
        try:
            return fn(timeout=max(0.0, deadline - time.monotonic()))
        except RetryableError as exc:
            delay = exc.retry_after or backoff_delay(attempt)
            last = attempt == max_attempts
            if last or time.monotonic() + delay >= deadline or not budget.can_retry():
                raise
            time.sleep(delay)

Rule 6: make retries visible

Retries hide failures by design, which means you can be one step away from an outage without seeing it. We export, per client and dependency:

• First attempts, retries, and retry budget exhaustion.
• Success on first attempt versus success after retry.

An alert fires when the retry rate exceeds 5 percent for 10 minutes. A dependency that only works because of retries is a dependency that is failing.

A reference configuration

What a typical internal HTTP client looks like after review:

Summary

• Retry only failures that can succeed on another attempt, and only repeat requests that are safe to repeat.
• Use exponential backoff with full jitter and a small maximum number of attempts.
• Cap total retry load with a budget, so a full outage adds little extra traffic.
• Retry at one layer of the call chain; retries multiply across layers.
• Keep retries within the request deadline, and monitor the retry rate as a health signal.

References: AWS Architecture Blog, Exponential Backoff and Jitter, gRPC retry design, RFC 9110, idempotent methods, Google SRE book, handling overload.