kubectl Unauthorized on a fresh Terraform-built EKS cluster: access entries explained

We built a new EKS cluster for the payments platform with Terraform from CI. terraform apply finished green: cluster, managed node group, VPC CNI add-on, all created. Then the follow-up step that installs our base Helm charts failed, and when platform engineers tried the cluster by hand, they got the same answer:

$ kubectl get nodes
E0802 09:14:27.118204   41822 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)

With verbose output the status is clearer:

$ kubectl get nodes -v=6
I0802 09:14:41.552310   41838 round_trippers.go:560] GET https://4C7E1A2B9D0F3E5A.gr7.eu-west-1.eks.amazonaws.com/api?timeout=32s 401 Unauthorized in 212 milliseconds

And from the CI job:

Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials

A 401 means the API server did not accept the identity at all. A 403 would have meant the identity was known but lacked permissions. That distinction drove the whole investigation.

Reproducing it

Every principal we tried failed: engineers using the PlatformAdmin permission set through IAM Identity Center, and the CI role gha-terraform-deploy that had created the cluster. Only the node group's EC2 role could talk to the API server, which is why the nodes had joined.

What we ruled out

• Stale kubeconfig. We regenerated it with aws eks update-kubeconfig --name payments-prod --region eu-west-1. Same 401.
• Wrong AWS identity locally. aws sts get-caller-identity returned the expected SSO session in the right account.
• Endpoint access. The cluster had a public endpoint restricted to our office and CI egress CIDRs, and the request reached the server (we got a 401, not a timeout).
• Token generation. aws eks get-token --cluster-name payments-prod returned a token; the problem was what the server mapped it to.

Root cause

We queried the cluster's access configuration and the access entries it actually had:

$ aws eks describe-cluster --name payments-prod --query 'cluster.accessConfig'
{
    "bootstrapClusterCreatorAdminPermissions": false,
    "authenticationMode": "API"
}

$ aws eks list-access-entries --cluster-name payments-prod --output text
ACCESSENTRIES   arn:aws:iam::111122223333:role/PlatformAdmin
ACCESSENTRIES   arn:aws:iam::111122223333:role/payments-prod-node-group-20260801

Three facts explained everything:

1. authenticationMode was API, so the aws-auth ConfigMap is ignored entirely. Only access entries authenticate IAM principals.
2. bootstrapClusterCreatorAdminPermissions was false, a deliberate hardening choice in our module. So the CI role that created the cluster got no entry and no admin rights.
3. The entry for humans pointed at role/PlatformAdmin, a role name someone typed into a variable. The real role behind the Identity Center permission set is generated, with a suffix and a path:

$ aws sts get-caller-identity --query Arn --output text
arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_PlatformAdmin_1a2b3c4d5e6f7a8b/dev-user

$ aws iam list-roles --path-prefix /aws-reserved/sso.amazonaws.com/ \
    --query "Roles[?starts_with(RoleName, 'AWSReservedSSO_PlatformAdmin_')].Arn" --output text
arn:aws:iam::111122223333:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_PlatformAdmin_1a2b3c4d5e6f7a8b

role/PlatformAdmin existed as an old, unused IAM role, so creating the access entry succeeded and nothing warned us. It simply matched nobody.

What is happening under the hood

EKS authenticates IAM principals with a presigned STS GetCallerIdentity token. The API server's authenticator resolves the token to the IAM role behind the session, then looks for a mapping. Historically that mapping lived in the aws-auth ConfigMap. Access entries, introduced in late 2023, move it into the EKS API, where it is managed with IAM permissions and Terraform resources instead of a ConfigMap edit inside the cluster.

The cluster's authenticationMode decides which source counts:

Moving forward along that table is allowed; moving back is not.

An access entry alone only establishes identity. Permissions come from access policies associated with the entry (AmazonEKSClusterAdminPolicy, AmazonEKSAdminPolicy, AmazonEKSEditPolicy, AmazonEKSViewPolicy), scoped to the cluster or to namespaces, or from Kubernetes RBAC bound to the entry's kubernetes_groups. No entry gives 401; an entry without policies or RBAC gives 403.

bootstrapClusterCreatorAdminPermissions is read only when the cluster is created. When true, EKS adds an admin access entry for whoever called CreateCluster. We had turned it off so that admin access is always explicit in code, which is a good practice, but then the code must include the CI role.

Environment

Minimal reproduction

This is the shape of the configuration that produced the bug, trimmed to the access-related parts:

variable "admin_role_arns" {
  type    = list(string)
  default = ["arn:aws:iam::111122223333:role/PlatformAdmin"]
}

resource "aws_eks_cluster" "this" {
  name     = "payments-prod"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.33"

  access_config {
    authentication_mode                         = "API"
    bootstrap_cluster_creator_admin_permissions = false
  }

  vpc_config {
    subnet_ids = var.private_subnet_ids
  }
}

resource "aws_eks_access_entry" "admin" {
  for_each      = toset(var.admin_role_arns)
  cluster_name  = aws_eks_cluster.this.name
  principal_arn = each.value
}

resource "aws_eks_access_policy_association" "admin" {
  for_each      = toset(var.admin_role_arns)
  cluster_name  = aws_eks_cluster.this.name
  principal_arn = each.value
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"

  access_scope {
    type = "cluster"
  }

  depends_on = [aws_eks_access_entry.admin]
}

Apply this as one role, then try kubectl as any role not in admin_role_arns, including the one that ran the apply. Every attempt returns 401.

Who is affected

Clusters created with authentication_mode = "API" or API_AND_CONFIG_MAP where creator bootstrap is disabled, which the popular community EKS module also defaults to in recent major versions. Teams using IAM Identity Center are especially exposed, because permission set roles have generated names that change if the permission set is recreated. Clusters still on CONFIG_MAP mode fail differently: there the usual mistake is an aws-auth entry with the role path left in.