Kubernetes RBAC "cannot list resource pods at the cluster scope" with a RoleBinding in place

The payments team deployed an internal status dashboard that showed their pods, deployments and recent events. It ran in the payments namespace with its own service account. They had created a Role and a RoleBinding granting read access to pods and deployments in their namespace. The dashboard showed an empty page, and its logs were full of:

pods is forbidden: User "system:serviceaccount:payments:status-dashboard" cannot list resource "pods"
in API group "" at the cluster scope

They asked the platform team for "more permissions". Before granting anything, we wanted to know exactly which request failed and why.

The RBAC objects

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: status-reader
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["pods", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: status-dashboard-reader
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: status-dashboard
    namespace: payments
roleRef:
  kind: Role
  name: status-reader
  apiGroup: rbac.authorization.k8s.io

These were correct for what they intended.

Asking the API server directly

kubectl auth can-i evaluates RBAC for any user, including a service account, using impersonation:

kubectl auth can-i list pods -n payments \
  --as=system:serviceaccount:payments:status-dashboard
# yes

kubectl auth can-i list pods --all-namespaces \
  --as=system:serviceaccount:payments:status-dashboard
# no

The permission worked inside the namespace. The dashboard was asking for something else.

What the client requested

The API server audit log, filtered for the service account, showed the request path:

"verb":"list","requestURI":"/api/v1/pods?labelSelector=team%3Dpayments&limit=500",
"user":{"username":"system:serviceaccount:payments:status-dashboard"},"responseStatus":{"code":403}

/api/v1/pods is the cluster-wide pods endpoint. A namespaced request would be /api/v1/namespaces/payments/pods. The dashboard's code used the Kubernetes client to list pods across all namespaces and filtered by a team=payments label:

# dashboard (excerpt)
pods = v1.list_pod_for_all_namespaces(label_selector="team=payments")

What we ruled out

• A missing or misnamed binding. can-i in the namespace said yes.
• Wrong service account in the pod. The error message named the expected account, and kubectl get pod -o jsonpath='{.spec.serviceAccountName}' confirmed it.
• Admission or policy engines. The response was an RBAC 403 from the API server, not a webhook denial, which has a different message format.

Root cause

A Role grants permissions only inside its namespace. A list across all namespaces is a cluster-scoped request, which can only be authorized by a ClusterRole bound with a ClusterRoleBinding. The dashboard made a cluster-scoped request, so the namespaced Role could never satisfy it, regardless of the label selector.

The four RBAC objects

The combination that surprises people: a RoleBinding can reference a ClusterRole, and then it grants those permissions only in the RoleBinding's namespace. That is the standard way to reuse common permission sets, such as the built-in view ClusterRole, per namespace.

Label selectors do not narrow authorization

RBAC authorizes by verb, resource, namespace and resource name. It does not look at label selectors or field selectors. A request for "all pods with label team=payments" is a request for all pods, as far as RBAC is concerned, and the server filters only after authorization.

Reading the error message precisely

The RBAC denial message contains everything needed, if you read it word by word:

A namespaced denial ends with in the namespace "payments" instead. "At the cluster scope" therefore means the client asked for every namespace, or for a cluster-scoped resource such as nodes. That single phrase would have pointed straight at the client code.

Environment

Who is affected

Any in-cluster tool written against "all namespaces" APIs and then deployed with namespace-scoped permissions: dashboards, operators installed in single-namespace mode, CI deploy jobs that run kubectl get pods -A, and controllers built with client libraries whose default informers watch every namespace. Operator frameworks usually have a setting to watch only specific namespaces; it must match the RBAC you grant.

Change the client, not the permissions

The dashboard only needed its own namespace. The fix was one line in the client:

- pods = v1.list_pod_for_all_namespaces(label_selector="team=payments")
+ pods = v1.list_namespaced_pod(namespace="payments")

The same for deployments and events. The namespace is read from the mounted service account file, so the same image works in any namespace:

NAMESPACE = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip()
pods = v1.list_namespaced_pod(namespace=NAMESPACE)

Reuse the built-in view role

We also replaced the custom Role with the built-in view ClusterRole, bound per namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: status-dashboard-view
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: status-dashboard
    namespace: payments
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

view grants read access to most namespaced resources and deliberately excludes Secrets.

When cross-namespace access is really needed

If a dashboard does need several namespaces, bind the ClusterRole with a RoleBinding in each of those namespaces and have the client query each namespace. Granting a ClusterRoleBinding to view would let the service account read every namespace, including other teams' configuration.

Alternatives and trade-offs

• A ClusterRoleBinding to a read-only ClusterRole. Quick, and what the team first asked for, but it exposes all namespaces to anyone who compromises the dashboard.
• An aggregated per-team API in front of the cluster. Overkill for one dashboard.

How to test the fix

kubectl auth can-i --list -n payments --as=system:serviceaccount:payments:status-dashboard | head
kubectl auth can-i list pods --all-namespaces --as=system:serviceaccount:payments:status-dashboard
# no  (and that is correct)
kubectl logs -n payments deploy/status-dashboard | grep -c forbidden
# 0

Confirming in production

The dashboard showed the team's pods and deployments. The audit log showed only namespaced requests under /api/v1/namespaces/payments/... with status 200, and no 403s for the service account.

Regression guard

• A CI check runs kubectl auth can-i assertions from a small YAML spec for each service account: what it must be allowed and what it must not.
• A policy alerts on new ClusterRoleBindings that bind to service accounts outside the platform namespaces.

Security implications

Resolving RBAC errors by widening permissions is the most common way clusters end up over-privileged. Here, the correct fix reduced the client's request scope instead. RBAC does not filter by labels, so a cluster-wide read grant would have exposed every namespace's pods, environment variables in pod specs and events. Keep Secrets out of read roles; view already does.

Prevention checklist

• Read RBAC errors carefully: "at the cluster scope" means a cluster-wide request.
• Use kubectl auth can-i --as to test permissions for service accounts.
• Make clients use namespaced API calls when they only need one namespace.
• Reuse ClusterRoles through RoleBindings rather than ClusterRoleBindings.
• Check the audit log for the exact request URI before granting anything.

References

• Kubernetes, Using RBAC Authorization: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
• kubectl auth can-i: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_auth/kubectl_auth_can-i/
• Kubernetes, Auditing: https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
• Kubernetes, RBAC good practices: https://kubernetes.io/docs/concepts/security/rbac-good-practices/