KEDA kept our SQS workers at zero replicas while the queue filled: operator identity and AWS auth

We moved an image-processing worker from a fixed three replicas to KEDA, so it could scale from zero to fifty based on the depth of its SQS queue. In staging, a quick test looked fine because the worker happened to be at one replica from a previous deploy. In production, the first big upload batch arrived on a Monday morning. Two hours later the queue held 40,000 messages and the Deployment had zero pods.

The ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: image-worker
  namespace: media
spec:
  scaleTargetRef:
    name: image-worker
  minReplicaCount: 0
  maxReplicaCount: 50
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/111122223333/image-jobs
        queueLength: "20"
        awsRegion: eu-west-1
        identityOwner: pod

What KEDA reported

kubectl get scaledobject image-worker -n media

NAME           SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   READY   ACTIVE   FALLBACK
image-worker   apps/v1.Deployment   image-worker      0     50    False   False    False

READY False was the clue we had missed. The conditions explained:

kubectl describe scaledobject image-worker -n media | sed -n '/Conditions/,/Events/p'

  Type    Status  Reason                    Message
  Ready   False   ScaledObjectCheckFailed   ScaledObject doesn't have correct scaleTargetRef specification
  ...
Events:
  Warning  KEDAScalerFailed  keda-operator  error getting metric: operation error SQS: GetQueueAttributes,
  https response error StatusCode: 403, api error AccessDenied: User:
  arn:aws:sts::111122223333:assumed-role/eks-node-group-role/i-0a1b2c3d is not authorized to perform:
  sqs:GetQueueAttributes on resource: arn:aws:sqs:eu-west-1:111122223333:image-jobs

The Ready message about scaleTargetRef was a red herring from an earlier typo we had already fixed; the event below it was the real error. KEDA called SQS as the node's IAM role, which had no SQS permissions.

What we ruled out

• The worker's own permissions. The worker's service account had an IAM role through EKS Pod Identity with full access to the queue. With zero replicas, the worker never ran, and its identity was irrelevant to the scaling decision.
• Queue URL or region. The error named the right queue ARN.
• KEDA's metrics server. The HPA KEDA created existed and showed <unknown> for the external metric, consistent with the scaler failing.

Root cause

KEDA's scaler runs inside the KEDA operator pod, not in the workload. With identityOwner: pod and no TriggerAuthentication, the aws-sqs-queue scaler used the default AWS credential chain of the KEDA operator pod. The operator's service account had no IAM role, so the SDK fell back to the EC2 instance metadata credentials of the node, which lacked sqs:GetQueueAttributes. KEDA could not read the queue depth, never activated, and the Deployment stayed at zero.

Who calls AWS in KEDA

When a workload is scaled to zero, only the operator can see the queue. So the operator must have a way to authenticate: its own identity, or credentials referenced through a TriggerAuthentication.

identityOwner and TriggerAuthentication

For AWS scalers, KEDA supports several authentication modes. The two relevant here:

• Operator identity. Give the KEDA operator's service account an IAM role (IRSA or Pod Identity) with read access to the queues. Simple, but the operator then has access to every queue in the policy.
• A TriggerAuthentication with podIdentity.provider: aws and a roleArn: the operator, using its own identity, assumes the specified role for that scaler. This lets you grant per-workload roles while the operator only needs sts:AssumeRole.

Environment

Step 1: an identity for the KEDA operator

We created an IAM role trusted by EKS Pod Identity and associated it with the KEDA operator's service account:

aws eks create-pod-identity-association \
  --cluster-name prod \
  --namespace keda \
  --service-account keda-operator \
  --role-arn arn:aws:iam::111122223333:role/keda-operator

The operator role can only assume scaler roles:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["sts:AssumeRole", "sts:TagSession"],
    "Resource": "arn:aws:iam::111122223333:role/keda-scaler-*"
  }]
}

The operator pod must be restarted after the association so it receives the credentials.

Step 2: a scaler role per workload

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["sqs:GetQueueAttributes"],
    "Resource": "arn:aws:sqs:eu-west-1:111122223333:image-jobs"
  }]
}

Its trust policy allows the keda-operator role to assume it.

Step 3: reference it from the ScaledObject

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: image-worker-sqs
  namespace: media
spec:
  podIdentity:
    provider: aws
    roleArn: arn:aws:iam::111122223333:role/keda-scaler-image-jobs
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: image-worker
  namespace: media
spec:
  scaleTargetRef:
    name: image-worker
  minReplicaCount: 0
  maxReplicaCount: 50
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: image-worker-sqs
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/111122223333/image-jobs
        queueLength: "20"
        awsRegion: eu-west-1

We removed identityOwner: pod, which only applies to the older pod identity modes and confused readers.

Step 4: block the node role fallback

The silent fallback to the node's credentials was the dangerous part. We set the instance metadata hop limit to 1 on all nodes, so pods cannot reach node credentials at all:

metadata_options {
  http_tokens                 = "required"
  http_put_response_hop_limit = 1
}

With that, a missing identity fails with a clear "no credentials" error instead of borrowing the node's permissions.

Alternatives and trade-offs

• Grant the operator SQS read directly. Fewer moving parts, but one identity can read every queue. Acceptable for small clusters with one team.
• minReplicaCount: 1. Hides the problem, since the worker always exists, but KEDA still cannot scale up beyond what the HPA's CPU metric suggests.

How to test the fix

kubectl rollout restart deployment keda-operator -n keda
kubectl get scaledobject image-worker -n media -w
# READY True, ACTIVE True once messages are queued

aws sqs send-message --queue-url https://sqs.eu-west-1.amazonaws.com/111122223333/image-jobs --message-body test
kubectl get deploy image-worker -n media -w     # scales from 0 to 1

Confirming in production

The ScaledObject became READY True. The Deployment scaled to 50 replicas within about two minutes and drained the 40,000-message backlog in 35 minutes, then returned to zero after the cooldown period. CloudTrail showed GetQueueAttributes calls from the keda-scaler-image-jobs role session.

Regression guard

• An alert on keda_scaler_errors_total and on ScaledObjects with Ready=False for more than five minutes.
• A queue age alarm (ApproximateAgeOfOldestMessage) independent of KEDA, so a broken scaler is noticed even if KEDA's own metrics fail.
• New ScaledObjects are tested in staging starting from zero replicas, not from an existing pod.

Security implications

The original setup worked partly by accident on other clusters, where the node role had broad permissions, meaning every pod on those nodes could use them. Giving the operator its own identity, using per-scaler roles with only GetQueueAttributes, and blocking pod access to node credentials reduced the blast radius considerably.

Prevention checklist

• Remember that KEDA scalers authenticate as the operator, not as the workload.
• Give the KEDA operator an explicit IAM identity and use TriggerAuthentication for per-workload roles.
• Block pod access to node instance credentials with a metadata hop limit of 1.
• Check READY and events on every new ScaledObject.
• Test scale-from-zero explicitly.

References

• KEDA AWS SQS Queue scaler: https://keda.sh/docs/latest/scalers/aws-sqs/
• KEDA AWS (EKS Pod Identity) authentication: https://keda.sh/docs/latest/authentication-providers/aws/
• Amazon EKS Pod Identity: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html
• EC2 instance metadata options: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-existing-instances.html