One availability zone failed and took all our replicas: topology spread constraints done right

On a Saturday afternoon, one availability zone in our region had a network disruption that lasted about 25 minutes. Most services degraded gracefully: they lost a third of their capacity and autoscaled. The search API went completely down. It ran three replicas on a cluster with nodes in three zones, and all three replicas were in the failing zone.

kubectl get pods -n search -l app=search-api -o wide

NAME                          READY   STATUS    NODE
search-api-74d8c9f6b5-4kqzt   1/1     Running   ip-10-20-12-88.eu-west-1.compute.internal
search-api-74d8c9f6b5-9mxlw   1/1     Running   ip-10-20-12-88.eu-west-1.compute.internal
search-api-74d8c9f6b5-tw2rc   1/1     Running   ip-10-20-13-201.eu-west-1.compute.internal

kubectl get nodes ip-10-20-12-88.eu-west-1.compute.internal ip-10-20-13-201.eu-west-1.compute.internal \
  -L topology.kubernetes.io/zone
# both: eu-west-1a

Two replicas on the same node, the third on another node in the same zone.

Why did the scheduler do that?

The Deployment had no affinity rules or spread constraints. It requested a lot of memory (6 Gi per pod for an in-memory index). When the pods were last rescheduled, after a node rotation, only nodes in eu-west-1a had enough free memory, because a large batch job was occupying the other zones.

What we ruled out

• Node selectors pinning a zone. None in the spec.
• A zonal volume. The search API was stateless; it loaded its index from object storage at startup.
• Scheduler misconfiguration. The default scheduler profile was in use.

Root cause

Kubernetes' default scheduling does prefer spreading pods of the same workload across nodes and zones, but only as a soft preference, weighed against other scores such as resource fit. When resources were tight, the soft preference lost, and nothing forced the replicas apart. The Deployment had redundancy in count, but not in failure domain.

Default spreading is a preference

Since Kubernetes 1.24, the scheduler applies built-in default topology spread constraints: a ScheduleAnyway preference across hostnames and zones. ScheduleAnyway means "prefer to spread, but place the pod even if it cannot be spread". It nudges, it does not guarantee.

topologySpreadConstraints

You can declare explicit spreading rules per workload:

Who is affected

Any multi-replica workload without explicit spread rules, especially ones with large resource requests that fit on few nodes, workloads rescheduled together during node rotations, and clusters where batch jobs temporarily fill some zones. Clusters with Karpenter or Cluster Autoscaler are not protected automatically: autoscalers add nodes where pending pods need them, but without a hard spread rule, no pod is ever pending for the empty zone.

Environment

Hard spread across zones, soft spread across nodes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: search-api
  namespace: search
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: search-api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          minDomains: 3
          labelSelector:
            matchLabels:
              app: search-api
          matchLabelKeys: ["pod-template-hash"]
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: search-api
          matchLabelKeys: ["pod-template-hash"]

What each choice does:

• Zone constraint DoNotSchedule with maxSkew: 1: with three replicas and three zones, each zone gets one. A pod that cannot be placed without breaking the rule stays Pending rather than piling into one zone, and a Pending pod triggers the autoscaler to add a node in the zone that needs it.
• minDomains: 3: without it, if one zone has no eligible nodes at all, the scheduler only counts the zones it sees and might place everything in two. With it, the scheduler treats a missing zone as having zero pods, which forces pods to wait for capacity there.
• Hostname constraint ScheduleAnyway: prefer different nodes, but do not block scheduling on it.
• matchLabelKeys: ["pod-template-hash"]: during a rolling update, count only pods of the new ReplicaSet, so old pods being terminated do not distort the spread.

Capacity to make the rule satisfiable

A hard constraint is only useful if capacity exists in every zone. We use one node group per zone with --balance-similar-node-groups in Cluster Autoscaler, and reserve headroom with a low-priority placeholder Deployment in each zone, so a new search pod can preempt it immediately instead of waiting minutes for a node.

PodDisruptionBudget

Spreading protects against zone failure; a PDB protects against voluntary disruptions taking down too many pods at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: search-api
  namespace: search
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: search-api

Alternatives and trade-offs

• podAntiAffinity with requiredDuringSchedulingIgnoredDuringExecution on zone. It works for "at most one per zone", but it cannot express "spread evenly" for more replicas than zones. Spread constraints are more flexible.

• More replicas. Six replicas spread softly will probably cover every zone, but "probably" is what failed us.

• Cluster-level default constraints. The scheduler configuration can set default spread constraints for all pods. On managed control planes you usually cannot change scheduler configuration, so we set the rule per workload through a shared Helm library chart instead.

How to test the fix

kubectl apply -f search-api.yaml
kubectl get pods -n search -l app=search-api -o wide \
  | awk 'NR>1{print $7}' | xargs -I{} kubectl get node {} -L topology.kubernetes.io/zone --no-headers \
  | awk '{print $NF}' | sort | uniq -c
#   1 eu-west-1a
#   1 eu-west-1b
#   1 eu-west-1c

To test the failure case, cordon every node in one zone on staging and delete that zone's pod. It should stay Pending, not move into another zone, and the autoscaler should add capacity in the cordoned zone's node group once the cordon is lifted.

Confirming in production

All three search pods ran in different zones. A month later, a planned game day blocked traffic to one zone for 15 minutes: search lost one replica, the other two absorbed the load, and the API stayed up with slightly higher latency.

Regression guard

• An admission policy requires a zone topologySpreadConstraint on Deployments with two or more replicas in production namespaces.
• A daily report lists workloads whose ready pods are concentrated in one zone.
• Game days include a zone-impairment scenario twice a year.

Security implications

None directly. Availability is part of security's CIA triad; a single zone failure making a customer- facing service unavailable is exactly the kind of risk redundancy is meant to remove.

Prevention checklist

• Do not rely on default spreading for availability; it is a preference.
• Use a zone spread constraint with DoNotSchedule, maxSkew: 1 and minDomains.
• Add matchLabelKeys: ["pod-template-hash"] for clean rolling updates.
• Ensure capacity per zone, with per-zone node groups or a zone-aware provisioner.
• Test zone failure regularly.

References

• Kubernetes, Pod Topology Spread Constraints: https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/
• Kubernetes, Assigning Pods to Nodes: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
• AWS, EKS best practices for reliability: https://docs.aws.amazon.com/eks/latest/best-practices/reliability.html
• Cluster Autoscaler FAQ: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md