StatefulSet pod stuck Pending with volume node affinity conflict after a node moved zones

We replaced a managed node group to move to a newer instance type. The new group was created, the old one drained and deleted, and everything rescheduled, except one pod of the Redis StatefulSet used for background job locks:

$ kubectl get pods -n jobs -l app=redis-locks
NAME            READY   STATUS    RESTARTS   AGE
redis-locks-0   1/1     Running   0          3m
redis-locks-1   1/1     Running   0          3m
redis-locks-2   0/1     Pending   0          3m

$ kubectl describe pod redis-locks-2 -n jobs | tail -5
Events:
  Type     Reason            Age   From               Message
  Warning  FailedScheduling  2m    default-scheduler  0/6 nodes are available: 2 Insufficient memory,
  4 node(s) had volume node affinity conflict. preemption: 0/6 nodes are available: 2 No preemption
  victims found for incoming pod, 4 Preemption is not helpful for scheduling.

Following the volume

kubectl get pvc data-redis-locks-2 -n jobs -o jsonpath='{.spec.volumeName}'
# pvc-3f8a1c2e-...
kubectl get pv pvc-3f8a1c2e-... -o yaml | yq '.spec.nodeAffinity'

required:
  nodeSelectorTerms:
    - matchExpressions:
        - key: topology.ebs.csi.aws.com/zone
          operator: In
          values:
            - eu-west-1a

The persistent volume could only be used by nodes in eu-west-1a. Where were the nodes?

kubectl get nodes -L topology.kubernetes.io/zone

NAME                          STATUS   ZONE
ip-10-20-11-34...             Ready    eu-west-1a
ip-10-20-12-201...            Ready    eu-west-1a
ip-10-20-21-77...             Ready    eu-west-1b
ip-10-20-22-15...             Ready    eu-west-1b
ip-10-20-31-90...             Ready    eu-west-1c
ip-10-20-32-6...              Ready    eu-west-1c

Two nodes in eu-west-1a, and both were full: "2 Insufficient memory". The other four were in the wrong zone for this volume.

What we ruled out

• A broken volume. The EBS volume was available in the EC2 console, in eu-west-1a, detached cleanly from the old node.
• The CSI driver. Other pods attached volumes normally; the driver pods were healthy.
• Missing StorageClass. The PVC was bound; the problem was placement, not provisioning.

Root cause

EBS volumes exist in exactly one availability zone and can only attach to instances in that zone. The persistent volume for redis-locks-2 was created in eu-west-1a when the pod first started there. After the node group replacement, the new nodes in eu-west-1a were already filled by other pods that had rescheduled first, so the scheduler could not place the pod anywhere it could use its volume. Nothing would add capacity in that zone, because the managed node group's autoscaling had no reason to scale a specific zone.

Zonal storage and scheduling

WaitForFirstConsumer solves the first placement: the volume follows the pod. It does not help later: once the volume exists, the pod must follow the volume, forever. StatefulSets keep the same PVC per ordinal, so redis-locks-2 is tied to eu-west-1a for its lifetime.

Why autoscalers can help, or not

Cluster Autoscaler and Karpenter both understand volume topology for pending pods. Cluster Autoscaler can only scale node groups that span or are pinned to the right zone; a single node group spanning three zones may add a node in the wrong one. Karpenter launches a node in the zone the volume requires. We used managed node groups with Cluster Autoscaler at the time, configured with one node group across all three zones, which is the setup the Cluster Autoscaler documentation warns against for zonal volumes.

Who is affected

Any StatefulSet or pod with a PVC backed by zonal block storage: EBS on AWS, zonal persistent disks on Google Cloud, and zonal Azure managed disks. Risk is highest during node group replacements, cluster upgrades that rotate nodes, and zone-level capacity shortages for a specific instance type. Workloads on network file systems such as EFS, which have no zone affinity, and stateless Deployments are not affected.

Environment

Immediate: make room in the right zone

We scaled the node group up by one, which happened to add a node in eu-west-1b, then by another, which landed in eu-west-1a. The pod scheduled. That confirmed the diagnosis but was not a fix.

Node groups per zone

For workloads with zonal volumes, the Cluster Autoscaler documentation recommends one node group per availability zone, with --balance-similar-node-groups so the groups grow together:

module "nodes" {
  for_each = toset(["eu-west-1a", "eu-west-1b", "eu-west-1c"])
  source   = "./modules/managed-node-group"

  name         = "general-${each.key}"
  subnet_ids   = [local.private_subnet_by_az[each.key]]
  min_size     = 2
  max_size     = 10
  desired_size = 2
}

--balance-similar-node-groups=true
--expander=least-waste

Now when redis-locks-2 is Pending because of eu-west-1a, the autoscaler scales the eu-west-1a group specifically.

Spread replicas across zones from the start

The StatefulSet now spreads replicas across zones, so each volume lives in a different zone and one zone's capacity problem affects at most one replica:

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: redis-locks

Make node replacements zone-aware

Our node group replacement runbook now creates new nodes in every zone before draining old ones, and drains one zone at a time, so pods with zonal volumes always find capacity in their zone.

Alternatives and trade-offs

• Karpenter provisions nodes in the zone a pending pod's volume requires, which removes this class of problem. We moved to it later.
• Regional or network storage (EFS, or a managed Redis) has no zone affinity. For lock data that can be rebuilt, ElastiCache would have avoided persistent volumes entirely.
• PriorityClass for StatefulSets, so they can preempt less important pods in the right zone.

How to test the fix

On staging, cordon and drain all nodes in one zone, then check that the autoscaler adds a node in that zone for the pending StatefulSet pod:

kubectl get nodes -l topology.kubernetes.io/zone=eu-west-1a -o name | xargs -n1 kubectl drain --ignore-daemonsets --delete-emptydir-data
kubectl get events -n kube-system --field-selector source=cluster-autoscaler -w

Confirming in production

During the next node group replacement, all three Redis pods rescheduled within two minutes, each in its own zone. Cluster Autoscaler logs showed it choosing the zone-specific node group for the pending pod.

Regression guard

• An alert on pods Pending for more than five minutes with a volume node affinity conflict message.
• A policy requires topology spread constraints on StatefulSets with persistent volumes.

Security implications

None directly. For availability, the fix ensures a single zone's capacity shortage cannot keep a stateful replica down, which matters for anything holding locks or quorum.

Prevention checklist

• Remember that EBS and similar block volumes are zonal for life.
• Use WaitForFirstConsumer and spread StatefulSet replicas across zones.
• With Cluster Autoscaler, use one node group per zone and balance them.
• Replace nodes zone by zone, adding capacity before draining.
• Prefer zone-independent storage for data that does not need block volumes.

References

• Kubernetes, Storage Classes, volume binding mode: https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode
• Amazon EBS CSI driver: https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html
• Cluster Autoscaler on AWS, common notes and gotchas: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md
• Kubernetes, Pod Topology Spread Constraints: https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/