Karpenter consolidation restarted our pods every few minutes: disruption budgets and do-not-disrupt

Two days after we upgraded Karpenter to the v1 API on our production EKS cluster, the data team complained that their report generation pods, which run for 20 to 40 minutes, kept being killed and restarted from the beginning. At the same time, the API teams noticed a steady stream of restarts across their deployments, never more than one or two pods at a time, but constant.

The pods were not crashing. Their events showed eviction:

$ kubectl get events -n reports --field-selector reason=Evicted --sort-by=.lastTimestamp | tail -3
12m   Normal   Evicted   pod/report-runner-7d9c6b-x2kfp   Evicted pod: Underutilized
7m    Normal   Evicted   pod/report-runner-7d9c6b-q8tnz   Evicted pod: Underutilized
3m    Normal   Evicted   pod/report-runner-7d9c6b-h4lwm   Evicted pod: Underutilized

What Karpenter was doing

Karpenter emits events on NodeClaims and nodes when it disrupts them:

kubectl get events -A --field-selector source=karpenter --sort-by=.lastTimestamp | tail -8

default  Normal  DisruptionLaunching  nodeclaim/general-7rq2x  Launching NodeClaim: Underutilized/Replace
default  Normal  DisruptionTerminating node/ip-10-20-41-77...   Disrupting Node: Underutilized/Replace
default  Normal  DisruptionTerminating node/ip-10-20-12-9...    Disrupting Node: Underutilized/Delete
default  Normal  DisruptionLaunching  nodeclaim/general-k9w4d  Launching NodeClaim: Underutilized/Replace

The controller logs counted 46 consolidation actions in the previous hour, on a cluster of about 30 nodes. Karpenter was repeatedly finding a cheaper arrangement, replacing nodes, and then, as traffic moved and pods rescheduled, finding another.

The NodePool after the upgrade

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 0s
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

What we ruled out

• Spot interruptions. Interruption events have a different reason and come from the interruption queue; these said Underutilized.
• Node expiry. expireAfter was 720 hours; disruption reasons would say Expired.
• Drift. No Drifted events; the AMI and node class had not changed.

Root cause

During the v1 migration, the old consolidationPolicy: WhenUnderutilized was renamed to WhenEmptyOrUnderutilized, and the migration also introduced consolidateAfter for this policy, which someone set to 0s while converting manifests. That told Karpenter to consider every node for consolidation immediately after any pod change. With no disruption budget, no PodDisruptionBudgets on most workloads and no protection on the batch pods, Karpenter consolidated as aggressively as the cluster allowed.

How consolidation decides

Consolidation looks for nodes it can delete (their pods fit elsewhere) or replace with a cheaper instance. When it acts, it cordons the node, evicts pods through the Kubernetes eviction API (which respects PodDisruptionBudgets) and terminates the instance. Evicted pods are rescheduled; that rescheduling changes utilisation on other nodes, which can make another consolidation look worthwhile.

consolidateAfter is how long a node must go without pod scheduling or removal before it becomes a consolidation candidate. A value of 0s means nodes are evaluated constantly, so normal pod churn turns into node churn.

Three layers of protection

We had none of them configured.

Why replacements chain

A single consolidation decision often looks reasonable on its own: three half-full m6i.2xlarge nodes can be replaced by two, saving money. The problem is the second-order effect. Evicted pods land on the remaining nodes, which changes their utilisation, which makes a different combination look cheaper a minute later. With spot instances in the mix, Karpenter may also replace a node with a cheaper instance type in another family, and the new node's shape changes the picture again. Without a settling delay and budgets, this feedback loop runs as fast as the controller can evaluate it.

Who is affected

Clusters that upgraded from the beta APIs and translated WhenUnderutilized with consolidateAfter: 0s, clusters with many small Deployments that scale frequently (each scale event resets the picture), and any workload with long-running pods that cannot resume after eviction: batch jobs, video transcoding, CI runners and connection-heavy services such as WebSocket gateways.

Environment

Step 1: give consolidation time to settle

spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 10m

Ten minutes without scheduling changes before a node is a candidate cut consolidation actions by about 80% on its own.

Step 2: add NodePool disruption budgets

Budgets cap disruption per NodePool. We allowed at most 10% of nodes to be disrupted at once, and none for underutilisation during business hours in our main region:

spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 10m
    budgets:
      - nodes: "10%"
      - nodes: "0"
        reasons: ["Underutilized"]
        schedule: "0 7 * * mon-fri"
        duration: 11h

During the scheduled window, empty nodes can still be removed and drifted nodes replaced, but Karpenter will not move running workloads just to save money. Outside it, consolidation proceeds at 10% at a time.

Step 3: PodDisruptionBudgets for every service

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
  namespace: orders
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: orders-api

With a PDB, Karpenter's evictions wait until replacement pods are ready elsewhere. We added a policy that requires a PDB for every Deployment with more than one replica.

Step 4: protect pods that must not be interrupted

The report pods do work that cannot resume. They now carry the annotation:

spec:
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"

Karpenter skips voluntary disruption of any node running such a pod until it finishes. Use it for finite work, not for long-running services, or nodes can never be consolidated or updated. For safety we also set terminationGracePeriod on the NodePool, so even protected nodes are eventually replaced for security updates.

Alternatives and trade-offs

• consolidationPolicy: WhenEmpty: only remove empty nodes. Very stable, but you pay for half-used nodes.
• A separate NodePool for batch work with WhenEmpty and its own budgets, so batch nodes drain naturally. We did this later for the report workload.

How to test the fix

kubectl get nodepool general -o jsonpath='{.spec.disruption}' | jq
kubectl get events -A --field-selector source=karpenter -w | grep -i disrupt

On staging, deploy a pod with the annotation and confirm its node is never chosen while the pod runs.

Confirming in production

Over the following week, consolidation actions dropped from about 46 per hour to 3-8 per hour, mostly at night. The report pods completed without eviction. Compute cost rose by about 4% compared with the aggressive configuration, which was still 22% below the pre-Karpenter baseline.

karpenter_voluntary_disruption_decisions_total{reason="underutilized"}  (rate per hour: 46 -> 5)

Regression guard

• A policy check rejects NodePools with consolidateAfter below 5m or without budgets.
• Admission policy requires a PDB for multi-replica Deployments.
• An alert fires when evictions with reason Underutilized exceed 20 per hour.

Security implications

None directly. Note that do-not-disrupt and zero budgets can delay node replacement for security patches if overused. We keep expireAfter and terminationGracePeriod set so every node is replaced within a bounded time regardless of annotations.

Prevention checklist

• Never set consolidateAfter: 0s on general-purpose NodePools.
• Define NodePool disruption budgets, including business-hours windows if needed.
• Give every multi-replica service a PodDisruptionBudget.
• Mark finite, non-resumable pods with karpenter.sh/do-not-disrupt.
• Watch disruption decision metrics after Karpenter upgrades.

References

• Karpenter disruption concepts: https://karpenter.sh/docs/concepts/disruption/
• Karpenter NodePools: https://karpenter.sh/docs/concepts/nodepools/
• Karpenter v1 migration guide: https://karpenter.sh/docs/upgrading/v1-migration/
• Kubernetes, Specifying a Disruption Budget: https://kubernetes.io/docs/tasks/run-application/configure-pdb/