CreateContainerConfigError: couldn't find key in Secret after a secret store rename

A Monday deploy of our subscription API stalled. The rollout created two new Pods, both stuck, while the eight old Pods kept serving traffic, so customers noticed nothing. Our deploy pipeline timed out after fifteen minutes and paged the on-call engineer.

$ kubectl -n subs get pods -l app=subs-api
NAME                        READY   STATUS                       RESTARTS   AGE
subs-api-6f8c9d7b5-4ldwq    1/1     Running                      0          3d
subs-api-6f8c9d7b5-9xkpz    1/1     Running                      0          3d
subs-api-7b44f6c8d9-hm2rt   0/1     CreateContainerConfigError   0          14m
subs-api-7b44f6c8d9-tq8vn   0/1     CreateContainerConfigError   0          14m

The Pod events were explicit:

Warning  Failed  12m (x61 over 14m)  kubelet  Error: couldn't find key DATABASE_URL in Secret subs/subs-api-env

Reproducing it

Any new Pod failed the same way, including a Pod from the old ReplicaSet when we deleted one to test. That was an important clue: the problem was not the new image or the new manifest. It was the Secret as it existed now. The running Pods were fine only because environment variables are resolved once, when a container is created.

What we ruled out

RBAC or a missing Secret. A missing Secret produces a different message (Error: secret "subs-api-env" not found). The Secret existed and the kubelet could read it; it was the key that was absent.

A typo in the Deployment. Git history showed the secretKeyRef for DATABASE_URL had not changed in months.

Node-level problems. Pods failed on three different nodes, and other workloads started normally on the same nodes.

Root cause

Listing the keys showed what the Secret actually contained:

kubectl -n subs get secret subs-api-env -o json | jq -r '.data | keys[]'

DATABASE_PASSWORD
DATABASE_USER
DB_URL
REDIS_URL

DATABASE_URL had become DB_URL. The Secret is not edited by hand; it is produced by External Secrets Operator from an ExternalSecret that pulls a JSON document from our cloud secret manager and exposes each property as a key. On Friday, a teammate cleaning up naming in the secret manager renamed the property. ESO refreshed on its one-hour interval, faithfully rewrote the Kubernetes Secret with the new key, and nothing failed until the next Pod creation, three days later.

What is happening under the hood

When the kubelet creates a container, it builds the environment from env and envFrom. For each valueFrom.secretKeyRef it reads the referenced Secret from its cache and looks up the key. If the key is missing and the reference is not marked optional: true, the kubelet refuses to create the container and reports CreateContainerConfigError. It retries on every Pod sync, which is why the event count climbs, but it will never succeed until the key appears.

Three behaviours combine into a delayed failure:

• Environment variables are copied at container start. Changing or breaking a Secret has no effect on running containers.
• ESO's refreshInterval keeps the Secret in sync with the provider, including removals. With the default creationPolicy: Owner and a dataFrom.extract, keys that disappear upstream disappear in the cluster.
• A rolling update with maxUnavailable: 0 keeps old Pods until new ones are Ready, so a broken new Pod stalls the rollout instead of taking the service down. Good for availability, but it hides the problem until a scaling event, a node drain or a restart needs a fresh Pod.

If a node had been drained that weekend, the old Pods would have been evicted and could not have come back. We were one routine maintenance window away from an outage.

Environment

Minimal reproduction

kubectl create secret generic demo-env --from-literal=DB_URL=postgres://demo@db.example.com/app
kubectl run demo --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"demo","image":"busybox:1.36","command":["sh","-c","echo $DATABASE_URL; sleep 3600"],"env":[{"name":"DATABASE_URL","valueFrom":{"secretKeyRef":{"name":"demo-env","key":"DATABASE_URL"}}}]}]}}'
kubectl get pod demo
kubectl describe pod demo | grep "couldn't find key"

The Pod shows CreateContainerConfigError immediately. Adding the key with kubectl patch makes the next kubelet retry succeed without recreating the Pod.

Who is affected

Any workload that reads Secret or ConfigMap keys by name, where the Secret is generated from an external source: External Secrets Operator, Secrets Store CSI driver with sync, Vault agent injectors writing Secrets, sealed secrets regenerated by another team. Also anyone who uses envFrom and then references a variable that silently stopped existing; envFrom does not fail on missing keys, so the application gets an empty variable instead.

The fix

The immediate fix was to restore the key the application expects. Rather than reverting the rename in the secret manager (other consumers had already moved to DB_URL), we mapped the property explicitly in the ExternalSecret so the Kubernetes key name is owned by our repo, not by whoever edits the provider:

 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: subs-api-env
   namespace: subs
 spec:
   refreshInterval: 1h
   secretStoreRef:
     kind: ClusterSecretStore
     name: cloud-secrets
   target:
     name: subs-api-env
-  dataFrom:
-    - extract:
-        key: prod/subs-api
+  data:
+    - secretKey: DATABASE_URL
+      remoteRef:
+        key: prod/subs-api
+        property: DB_URL
+    - secretKey: DATABASE_USER
+      remoteRef:
+        key: prod/subs-api
+        property: DATABASE_USER
+    - secretKey: DATABASE_PASSWORD
+      remoteRef:
+        key: prod/subs-api
+        property: DATABASE_PASSWORD
+    - secretKey: REDIS_URL
+      remoteRef:
+        key: prod/subs-api
+        property: REDIS_URL

With explicit data entries, a renamed or deleted property makes the ExternalSecret itself fail to sync (SecretSyncedError) and ESO keeps the last good Secret instead of publishing one without the key. The failure moves from "next Pod start, days later" to "next refresh, with an alert".

We forced a refresh rather than waiting for the interval:

kubectl -n subs annotate externalsecret subs-api-env force-sync=$(date +%s) --overwrite
kubectl -n subs get externalsecret subs-api-env

The stuck Pods picked up the key on the kubelet's next retry and became Ready without a redeploy.

Alternatives and trade-offs

• optional: true on the key reference. Lets the container start with the variable unset. Only correct for genuinely optional settings; for a database URL it converts a loud failure into a confusing runtime crash.
• Mount the Secret as a volume. Missing keys in a volume mount do not block container creation, and files update in place. Again, that hides required keys, so we only use volumes for things the app re-reads, like TLS certificates.
• Keep dataFrom.extract and add validation. Works if the provider document is owned by the same team. Explicit mapping was simpler for a shared secret.

How to test the fix

We added a check that runs before every rollout and compares the keys the Deployment references with the keys the live Secret has:

#!/usr/bin/env bash
set -euo pipefail
ns=$1 deploy=$2
kubectl -n "$ns" get deploy "$deploy" -o json \
  | jq -r '.spec.template.spec.containers[].env[]? | select(.valueFrom.secretKeyRef) | select(.valueFrom.secretKeyRef.optional != true) | "\(.valueFrom.secretKeyRef.name) \(.valueFrom.secretKeyRef.key)"' \
  | sort -u | while read -r secret key; do
      if ! kubectl -n "$ns" get secret "$secret" -o json | jq -e --arg k "$key" '.data | has($k)' >/dev/null; then
        echo "missing key $key in secret $secret"; exit 1
      fi
    done
echo "all secret keys present"

Running it against the broken state printed missing key DATABASE_URL in secret subs-api-env and exited 1; after the fix it printed all secret keys present.

Confirming in production

kubectl -n subs rollout status deploy/subs-api --timeout=5m
kubectl -n subs get externalsecret subs-api-env -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

The rollout completed, and the ExternalSecret reported True. We then deleted one old Pod on purpose to prove a fresh Pod could start from the current Secret, which it did in nine seconds.

Regression guard

• The key check above runs in the deploy pipeline before kubectl apply, using the rendered manifest instead of the live Deployment.
• An alert on ESO's sync status metric (externalsecret_status_condition with condition="Ready" and status="False") for more than ten minutes.
• An alert on any Pod in CreateContainerConfigError for more than two minutes, from kube-state-metrics kube_pod_container_status_waiting_reason.
• A weekly job that restarts one replica of each critical Deployment, so "only works because it never restarted" cannot last for days.

Security implications

Be careful while debugging: kubectl get secret -o yaml prints every value base64-encoded, which is not encryption, into your terminal and possibly your shell history or incident notes. List keys only, as we did with jq 'keys'. The fix narrows the Secret to the four keys the service uses instead of every property in the provider document, which reduces what a compromised Pod can read. Make sure RBAC to read Secrets in the namespace stays limited to the service account and operators who need it.

Prevention checklist

• Map external secret properties to Kubernetes keys explicitly for anything the app requires.
• Alert on secret sync failures, not just on Pod failures.
• Validate referenced Secret keys before rollout.
• Restart a replica regularly so stale-but-running Pods do not mask broken config.
• Inspect Secret key names, never dump values, when troubleshooting.

References

• Kubernetes documentation, Secrets: https://kubernetes.io/docs/concepts/configuration/secret/
• Kubernetes documentation, Distribute Credentials Securely Using Secrets: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/
• External Secrets Operator documentation, ExternalSecret: https://external-secrets.io/latest/api/externalsecret/
• External Secrets Operator documentation, Metrics: https://external-secrets.io/latest/api/metrics/