AccessDenied after migrating from IRSA to EKS Pod Identity: agent, SDK versions and trust policy

IAM Roles for Service Accounts (IRSA) had served us well, but every new cluster meant updating trust policies in dozens of roles with the cluster's OIDC provider. EKS Pod Identity removes that: the trust policy trusts the EKS service, and associations live in the EKS API. We planned the migration service by service, removing the IRSA annotation and adding a Pod Identity association.

Nine services moved without incident. Three did not.

Service 1: invoice-renderer, AccessDenied on S3

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation:
User: arn:aws:sts::111122223333:assumed-role/prod-general-nodes/i-0f1e2d3c4b5a69788 is not authorized to
perform: s3:PutObject on resource: "arn:aws:s3:::acme-invoices/2025/03/inv-81822.pdf"

The error named the node group role, not the service's role. The pod had no Pod Identity credentials and fell back to instance metadata.

kubectl exec -n billing deploy/invoice-renderer -- env | grep -E 'AWS_CONTAINER|AWS_ROLE|AWS_WEB'

Nothing. When Pod Identity is working, the EKS webhook injects AWS_CONTAINER_CREDENTIALS_FULL_URI and AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE into the pod. Those variables were missing because the pods had been created before the association existed; associations only apply to pods created afterwards. A rollout restart fixed this one service. But the restart revealed a second problem on some nodes.

Service 1, continued: credentials URL unreachable

After the restart, half of the pods worked. The other half logged:

botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from container-role:
Error retrieving metadata: Received error when attempting to retrieve container metadata:
Could not connect to the endpoint URL: "http://169.254.170.23/v1/credentials"

All failing pods ran on a node group created recently by another team:

kubectl get pods -n kube-system -l app.kubernetes.io/name=eks-pod-identity-agent -o wide

The agent DaemonSet had no pods on that node group: it had a taint the agent did not tolerate.

Service 2: ledger-sync, still using IRSA credentials

This Java service kept working, but CloudTrail showed it still assuming its old IRSA role through AssumeRoleWithWebIdentity. The Helm chart had a default eks.amazonaws.com/role-arn annotation that our change did not remove for this service. With both mechanisms present, the AWS SDK credential chain picks the web identity provider before the container credential provider.

Service 3: legacy-exporter, AccessDenied on sts

A Go service built two years earlier failed with:

operation error STS: GetCallerIdentity, get identity: get credentials: failed to refresh cached
credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404

Its AWS SDK for Go v2 version predated support for the Pod Identity credential endpoint. It ignored the injected variables, fell through to IMDS, and our hop limit of 1 correctly blocked node credentials.

Root causes

• Pods created before the association, and the Pod Identity agent missing on a tainted node group.
• An IRSA annotation left in place, which takes precedence in the SDK credential chain.
• An AWS SDK too old to support the container credentials endpoint used by Pod Identity.

How Pod Identity delivers credentials

1. You create an association: namespace + service account + IAM role.
2. When a pod using that service account is created, the EKS mutating webhook adds environment variables pointing to the agent endpoint and mounts a projected token.
3. The Pod Identity agent, a DaemonSet on every node listening on 169.254.170.23, exchanges the token with the EKS Auth API for temporary role credentials.
4. The AWS SDK's container credential provider calls the agent.

Every step can fail independently: the association, the webhook timing (pod creation), the agent on the node, the SDK's support for the provider.

Credential precedence matters during migration

The default credential chain in AWS SDKs checks, roughly: explicit environment keys, web identity token (IRSA), shared config, container credentials (Pod Identity and ECS), then instance metadata. If a pod has both an IRSA annotation and a Pod Identity association, the SDK uses IRSA. That is convenient for a safe migration, but you must remove the annotation to complete it.

Minimum SDK versions

AWS publishes the minimum SDK version for each language that supports Pod Identity. Older SDKs do not read AWS_CONTAINER_CREDENTIALS_FULL_URI with the link-local agent address. Check every service's SDK version before migrating; Go, Java and Python services built years ago are the usual suspects.

Environment

Agent on every node

aws eks update-addon --cluster-name prod --addon-name eks-pod-identity-agent \
  --configuration-values '{"tolerations":[{"operator":"Exists"}]}'
kubectl get ds eks-pod-identity-agent -n kube-system
# DESIRED and READY now equal to the node count

A toleration for all taints is appropriate for a node-level credential agent; it must run wherever pods can run.

Restart pods after creating associations

Our migration script now creates the association, then runs kubectl rollout restart for the workload, then checks the environment variables in a new pod:

kubectl exec -n billing deploy/invoice-renderer -- printenv AWS_CONTAINER_CREDENTIALS_FULL_URI
# http://169.254.170.23/v1/credentials

Remove IRSA annotations

 serviceAccount:
   create: true
-  annotations:
-    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/ledger-sync-irsa
+  annotations: {}

After the restart, CloudTrail showed AssumeRoleForPodIdentity events for the new role, and the old IRSA role's last-used date stopped updating. We deleted the IRSA roles a month later.

Upgrade the SDK

The Go service was rebuilt with a current aws-sdk-go-v2 release. One minor code change was needed for a deprecated config option; no credential code changed, because the default chain picks up Pod Identity automatically.

The trust policy

One role we created by hand failed with AccessDenied on AssumeRoleForPodIdentity. Pod Identity tags sessions with cluster, namespace and service account, so the trust policy must allow sts:TagSession:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": ["sts:AssumeRole", "sts:TagSession"]
  }]
}

Alternatives and trade-offs

• Stay on IRSA. It works and supports more scenarios, such as cross-account OIDC federation and clusters outside EKS. Pod Identity is simpler for EKS-only setups.
• Run both permanently. Possible, but the precedence rule makes it confusing to know which role is in use.

How to test the fix

kubectl run whoami -n billing --rm -it --image=amazon/aws-cli \
  --overrides='{"spec":{"serviceAccountName":"invoice-renderer"}}' -- sts get-caller-identity
# "Arn": "arn:aws:sts::111122223333:assumed-role/invoice-renderer/eks-prod-invoice-r-..."

Confirming in production

All twelve services showed their Pod Identity role in sts get-caller-identity checks, CloudTrail showed no AssumeRoleWithWebIdentity calls from the cluster after the cleanup, and no requests came from the node group roles.

Regression guard

• An admission policy rejects service accounts that have the IRSA annotation in namespaces marked as migrated.
• The node group module adds the agent toleration check to its tests.
• A dependency check flags AWS SDK versions below the Pod Identity minimum.

Security implications

The fallback to node credentials in the first failure was the real risk: on a cluster without a metadata hop limit, the pod would have silently run with the node role's permissions. Our hop limit turned that into a visible failure for the Go service. Session tags from Pod Identity also enable attribute-based policies, for example limiting S3 access by the kubernetes-namespace tag.

Prevention checklist

• Ensure the Pod Identity agent runs on every node group, including tainted ones.
• Restart workloads after creating associations.
• Remove IRSA annotations to finish each migration.
• Check AWS SDK minimum versions per service before migrating.
• Include sts:TagSession in Pod Identity trust policies.

References

• Amazon EKS Pod Identity: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html
• Pod Identity supported SDK versions: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html
• Setting up the Pod Identity agent: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html
• IAM roles for service accounts: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
• AWS SDKs, container credential provider: https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html