Surviving traffic spikes: load balancing algorithms, autoscaling lag and load shedding

Our retail client runs flash sales announced by push notification. At 10:00 sharp, traffic goes from about 1,500 requests per second to 12,000 in roughly 40 seconds, holds for ten minutes, then decays. The first time it happened on our new platform, the autoscaler started adding capacity at 10:02, new pods were serving at 10:05, and by then the checkout service had been returning errors for four minutes.

This guide is what we changed after that sale. It is about the whole system rather than one component: how the load balancer spreads requests, why autoscaling is always late for sharp spikes, what to queue instead of serve, and how to reject excess load deliberately so the requests you do accept succeed.

Why spikes break systems that handle the same average load

A system sized for 12,000 requests per second survives a spike to 12,000. Ours was sized for 3,000 with autoscaling, which is reasonable economics, and the failure was in the transition. Three effects stack during a fast ramp:

• Queueing explodes non-linearly. As utilisation approaches 100 percent, wait time grows sharply. A server at 95 percent busy has queues many times longer than at 80 percent.
• Timeouts become retries. Clients and upstream services time out and retry, multiplying offered load exactly when there is least capacity.
• Cold capacity is slow capacity. New instances start with empty caches, cold connection pools and, for JVM or JIT runtimes, unoptimised code.

So the goal is not only "more servers". It is to keep each server below the knee of its latency curve during the ramp, and to decide what happens to requests that do not fit.

Choosing a load balancing algorithm

The algorithm matters most when backends are uneven: some pods are new and cold, some requests are slow, some instances share a noisy host.

Our checkout API ran behind an AWS Application Load Balancer using round robin. During the spike, new pods that were still warming received the same share as warm ones and became the slowest in the pool. Switching the target group to least outstanding requests let warm pods take more of the load while new ones caught up:

aws elbv2 modify-target-group-attributes \
  --target-group-arn "$TG_ARN" \
  --attributes Key=load_balancing.algorithm.type,Value=least_outstanding_requests

The alternative on round-robin groups is slow start (slow_start.duration_seconds), which ramps a new target's share over a window. The ALB does not allow slow start together with least outstanding requests, so pick one per target group; we used slow start on round-robin groups for static content and least outstanding requests on the API group.

For our internal services behind nginx, the equivalent is:

upstream inventory {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

keepalive matters during spikes: without it, every proxied request opens a new TCP connection to the backend, and the extra handshakes burn CPU on both sides.

One caveat for "least" algorithms: a backend that fails fast looks idle. If a pod returns 500 in two milliseconds, least outstanding requests sends it more traffic. Pair these algorithms with health checks and passive failure detection so failing backends are removed.

Why autoscaling is late, and how to close the gap

Our timeline from the first sale, reconstructed from metrics:

Every layer added delay: metric collection, the scaling decision, node provisioning, image pull, app start and readiness. Reactive scaling cannot beat a 40-second ramp. We used three tactics.

Pre-scale for known events. Sales are scheduled, so we scale up at 09:45 and back down after. A scheduled job raises the HPA's minReplicas:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout }
  minReplicas: 12
  maxReplicas: 120
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 55 } }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - { type: Percent, value: 100, periodSeconds: 15 }
    scaleDown:
      stabilizationWindowSeconds: 600

kubectl patch hpa checkout -p '{"spec":{"minReplicas":60}}'

Keep spare node capacity. Pods that wait for nodes are the slowest part. We run a low-priority placeholder deployment that reserves a few nodes' worth of resources; real pods preempt it instantly, and the cluster autoscaler replaces the placeholders in the background.

Scale on a leading signal. CPU rises after requests queue. Requests per second per pod, or in-flight requests, rise first. We target 150 requests per second per pod via a custom metric, which reacts earlier than CPU. A lower CPU target (55 percent rather than 80) also buys headroom while new capacity arrives.

Queue what does not need an immediate answer

Not every request in a spike needs to finish synchronously. During a sale, order confirmation emails, loyalty point updates, analytics events and stock sync with the warehouse can wait a minute. We moved them behind a queue so the API returns as soon as the order is committed, and workers drain the backlog at a rate the downstream systems can sustain.

The queue absorbs the spike and turns it into a longer, flatter load. It also isolates failures: if the email provider slows down, checkout does not. The trade-off is eventual consistency, so the UI says "confirmation on its way" rather than showing points instantly.

Shed load before it sheds you

When offered load exceeds capacity even after all of this, something has to give. The worst outcome is accepting everything and timing out on everything. The better outcome is rejecting some requests quickly so the rest complete.

We added three layers:

• Edge concurrency limits. nginx in front of checkout caps concurrent checkout connections per nginx instance, sized from load tests, and excess requests get an immediate 503 with Retry-After instead of waiting in an unbounded queue.
• Priority. Checkout and payment routes have reserved capacity; browsing and search are shed first. A user who cannot load recommendations is fine; a user who cannot pay is not.
• Bounded queues in the app. Each pod has a fixed-size worker pool and a short request queue. When the queue is full, the request is rejected in microseconds, which keeps latency for accepted requests stable.

limit_conn_zone $server_name zone=checkout_conn:10m;

location /api/checkout/ {
    limit_conn checkout_conn 800;
    limit_conn_status 503;
    proxy_pass http://checkout;
}

The client side cooperates: the app shows a "high demand, retrying" state and retries with backoff on 503 rather than hammering.

Caching and the CDN

The cheapest request is the one that never reaches the origin. Product pages and images for the sale are pushed to the CDN and pre-warmed by requesting them shortly before 10:00. Inventory counts are served from a short-lived cache with a 2-second TTL, so thousands of users looking at the same product generate one database read per two seconds instead of thousands.

What the next sale looked like

With pre-scaling at 09:45, least outstanding requests on the API, a custom request-rate metric, queued side effects and priority shedding, the next sale peaked at 14,500 requests per second. Checkout p99 latency stayed under 450 ms. The shedding layer rejected 2.3 percent of search requests for about 90 seconds and no checkout requests. Autoscaling still fired, but it added capacity on top of a system that was already coping.

Common mistakes

Key takeaways

• Spikes fail systems during the transition, not at the steady peak; plan for the ramp.
• Pick a load balancing algorithm that sees backend load when backends are uneven.
• Autoscaling is always late for sharp spikes: pre-scale, keep warm capacity, use leading metrics.
• Push deferrable work to queues so the synchronous path stays small.
• Shed low-priority load early and fast so critical requests succeed.

References

• AWS Elastic Load Balancing docs, Target group routing algorithms: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#modify-routing-algorithm
• Kubernetes docs, Horizontal Pod Autoscaling and scaling behavior: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
• Kubernetes Cluster Autoscaler FAQ, overprovisioning: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md
• nginx docs, ngx_http_upstream_module: https://nginx.org/en/docs/http/ngx_http_upstream_module.html
• nginx docs, ngx_http_limit_conn_module: https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html