Every deploy blocked after rotating our cosign key: Kyverno verifyImages pinned the old key

Our supply chain policy is simple: every image running in production must be signed by our CI pipeline with cosign, and Kyverno verifies the signature at admission. It had worked for a year. As part of an annual key rotation, the security team generated a new key pair, stored the private key in the CI system's KMS, and updated the pipeline to sign with it. The first release after the change failed to deploy:

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:

resource Deployment/orders/orders-api was blocked due to the following policies

verify-image-signatures:
  autogen-verify-signature: 'failed to verify image registry.example.com/orders-api:2.41.0:
  .attestors[0].entries[0].keys: no matching signatures: invalid signature when validating ASN.1 encoded
  signature'

Then it got worse. A node was replaced by the autoscaler, and pods that had been running fine for weeks could not be recreated, because ReplicaSets create new pods, and new pods go through admission too.

Warning  FailedCreate  replicaset/payments-api-7c9d8  Error creating: admission webhook
"mutate.kyverno.svc-fail" denied the request: ... no matching signatures

For those older images, the error was different: they were signed with the old key, which the policy no longer trusted, because someone had replaced the public key in the policy at the same time.

The policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 15
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["orders", "payments", "billing"]
      verifyImages:
        - imageReferences: ["registry.example.com/*"]
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...(new key)...
                      -----END PUBLIC KEY-----

What we ruled out

• Kyverno failing open or closed due to timeouts. The error was an explicit signature mismatch, not a webhook timeout.
• Registry access. Kyverno fetched the signatures; it reported that they did not match.
• Tag versus digest confusion. Kyverno resolves tags to digests and verifies the digest; the signature existed for that digest.

Root cause

The rotation was done in one step on both sides: the pipeline started signing with the new key, and the policy was changed to trust only the new key. Images already deployed were signed only with the old key. Any pod created from them, by a rollout, a scale-up or a node replacement, failed verification. And the first new image failed too, for a second reason we found later: the CI job had signed it with the new key but pushed the signature before a retagging step, so the signature was attached to a different digest. Two problems, one outage.

Admission happens on every pod creation

Kyverno's verifyImages rule runs at admission for every pod creation, not just at deploy time. Rescheduling, autoscaling, node maintenance and crash recovery all create pods. A policy change that invalidates the signatures of running images is therefore a time bomb: nothing breaks until the next pod needs to be created, and then it breaks everywhere.

Attestors and entries

In a verifyImages rule, attestors is a list of sets, and each set has entries and a count:

• Each attestor set must be satisfied (logical AND across sets).
• Within a set, count entries must verify; if count is omitted, all entries must verify.
• With count: 1, any one of the entries is enough (logical OR).

So to trust either the old or the new key during a rotation, put both keys in one attestor set with count: 1.

Environment

Emergency recovery

We needed pods to start again quickly, without disabling verification entirely. We changed the policy to trust both keys:

verifyImages:
  - imageReferences: ["registry.example.com/*"]
    attestors:
      - count: 1
        entries:
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                ...(old key)...
                -----END PUBLIC KEY-----
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                ...(new key)...
                -----END PUBLIC KEY-----

Rescheduled pods started within a minute. For the new release, we re-ran the signing step after the final push, which attached the signature to the correct digest:

cosign sign --yes --key awskms:///alias/cosign-prod-2025 \
  registry.example.com/orders-api@sha256:7b1f0c...
cosign verify --key cosign-prod-2025.pub registry.example.com/orders-api@sha256:7b1f0c...

The staged rotation we use now

Key rotation is now a four-step process, each step deployed separately:

1. Trust both keys. Add the new public key to the policy as a second entry with count: 1. Nothing is signed with it yet; this only prepares verification.
2. Sign with the new key. Switch CI to the new private key. New images carry new signatures, which the policy already trusts.
3. Re-sign images still in use. A script lists every image digest currently running in the cluster, plus the last N releases of each service kept for rollback, and signs them with the new key. cosign supports multiple signatures on the same digest, so the old signature stays.
4. Retire the old key. After a waiting period, remove the old key from the policy. Before doing so, verify that every running digest has a valid new-key signature.

# step 3: re-sign running images
kubectl get pods -A -o jsonpath='{range .items[*]}{range .status.containerStatuses[*]}{.imageID}{"\n"}{end}{end}' \
  | sed 's|^docker-pullable://||' | grep '^registry.example.com/' | sort -u \
  | xargs -I{} cosign sign --yes --key awskms:///alias/cosign-prod-2025 {}

# step 4 check: every running image verifies with the new key
... | xargs -I{} sh -c 'cosign verify --key cosign-prod-2025.pub {} > /dev/null || echo "MISSING {}"'

Sign by digest, after the final push

The CI fix for the second problem: sign the digest output by the push step, never a tag:

digest=$(docker buildx imagetools inspect registry.example.com/orders-api:2.41.0 --format '{{json .Manifest.Digest}}' | tr -d '"')
cosign sign --yes --key awskms:///alias/cosign-prod-2025 "registry.example.com/orders-api@${digest}"

Alternatives and trade-offs

• Keyless signing with Sigstore (OIDC identity from CI, Fulcio certificates, Rekor transparency log) removes long-lived keys and their rotation entirely. The policy verifies the identity of the signer, such as the CI workflow, instead of a key. We plan to move to it; it needs network access to the Sigstore services or a private instance.
• Audit mode during changes reports violations without blocking. Useful for testing a new policy, dangerous as a habit.

How to test the fix

On staging, run the rotation steps and, after each one, delete a pod from every workload to force recreation through admission:

kubectl delete pod -n orders -l app=orders-api --wait=false
kubectl get events -n orders --field-selector reason=FailedCreate -w

Confirming in production

After the emergency change all pods started. Over the next two weeks we completed steps 3 and 4; the verification check reported zero running digests without a new-key signature before the old key was removed. A forced restart of every production Deployment after step 4 completed without admission errors.

Regression guard

• Policy changes to verifyImages go through a pipeline that runs the "every running digest verifies" check against the proposed policy before merge.
• CI fails if cosign verify on the pushed digest fails, so a misattached signature is caught at build time.
• An alert on Kyverno admission denials above a small threshold per minute.

Security implications

The outage tempted people to switch the policy to Audit or delete it, which would have removed the supply chain control entirely. Trusting two keys temporarily keeps the control in force. The old key remains a risk until it is removed from the policy, so the rotation has a deadline, and the old private key was destroyed after step 4.

Prevention checklist

• Rotate keys in stages: trust both, sign with new, re-sign running images, remove old.
• Use one attestor set with count: 1 to trust either key.
• Sign the digest after the final push, and verify in CI.
• Remember that admission runs on every pod creation, not only on deploys.
• Consider keyless signing to remove key rotation from the picture.

References

• Kyverno, Verify Images Rules: https://kyverno.io/docs/writing-policies/verify-images/
• Kyverno, Sigstore verification: https://kyverno.io/docs/writing-policies/verify-images/sigstore/
• Sigstore cosign, signing containers: https://docs.sigstore.dev/cosign/signing/signing_with_containers/
• Sigstore, keyless signing overview: https://docs.sigstore.dev/cosign/signing/overview/