SSM Session Manager stopped working after enabling KMS encryption: missing kms:Decrypt

Our audit plan included an item that sounded harmless: "Encrypt Session Manager session data with a customer managed KMS key." We use AWS Systems Manager Session Manager instead of SSH for all access to our EC2 fleet, and the change was one setting in the Session Manager preferences. Ten minutes after an engineer applied it, the on-call channel filled with messages: nobody could open a session to any instance.

From the CLI the error was:

$ aws ssm start-session --target i-0abc123def4567890

An error occurred (TargetNotConnected) when calling the StartSession operation:
i-0abc123def4567890 is not connected.

Some engineers, on a newer CLI version, saw the session begin and then close immediately:

Starting session with SessionId: alice@example.com-0f1e2d3c4b5a69788

SessionId: alice@example.com-0f1e2d3c4b5a69788 :
----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on instance
i-0abc123def4567890 with status 2 error: Failed to process action KMSEncryption: ...

The instances showed as online in Fleet Manager. Run Command still worked. Only interactive sessions were broken.

Narrowing it down

The SSM Agent log on an instance reached through the serial console was the most useful source:

sudo tail -n 50 /var/log/amazon/ssm/amazon-ssm-agent.log

ERROR [SessionManager] ... KMS Decrypt failed: AccessDeniedException: User:
arn:aws:sts::111122223333:assumed-role/app-ec2-role/i-0abc123def4567890 is not authorized to perform:
kms:Decrypt on resource: arn:aws:kms:eu-west-1:111122223333:key/7d1e... because no identity-based policy
allows the kms:Decrypt action

On instances in our fully private subnets, the error was different: the agent timed out reaching kms.eu-west-1.amazonaws.com.

What we ruled out

• The SSM Agent version. All instances ran an agent version that supports KMS encryption for sessions.
• The engineers' own permissions. Their roles already had kms:GenerateDataKey on the key; the error came from the instance side.
• Network to SSM endpoints. Run Command worked, so the ssm, ssmmessages and ec2messages endpoints were reachable.

Root cause

When Session Manager encryption with KMS is enabled, both ends of the session use the key: the client user needs permission to generate a data key, and the instance's IAM role needs kms:Decrypt on the same key. Our instance profile role had no KMS permissions, the key policy did not delegate to IAM in a way that covered it, and private subnets had no VPC endpoint for KMS, so even with permission the agent could not reach the service.

How Session Manager uses the key

With KMS encryption enabled in the session preferences:

1. The client (AWS CLI with the Session Manager plugin, or the console) calls KMS to generate a data key for the session, which requires kms:GenerateDataKey for the user.
2. The encrypted data key is passed to the SSM Agent on the instance.
3. The agent calls KMS Decrypt using the instance profile credentials to obtain the plaintext data key.
4. Session traffic is then encrypted with that data key, on top of the TLS already used by the channel.

If step 3 fails, the handshake fails and the session ends. Older CLI and plugin versions surface it as a generic "not connected" error.

KMS authorization has two halves

For a customer managed key, a principal can use the key only if the key policy allows it, either directly or by delegating to IAM in the account, and, when delegated, an IAM policy attached to the principal allows the action. Our key policy contained the default statement that enables IAM policies for the account root, which is a delegation, so the missing piece was the IAM permission on the role.

Environment

Step 1: allow the instance role to decrypt with the key

We added a scoped inline policy to the instance role, managed in Terraform:

data "aws_iam_policy_document" "ssm_session_kms" {
  statement {
    sid       = "DecryptSessionDataKey"
    actions   = ["kms:Decrypt"]
    resources = [aws_kms_key.ssm_sessions.arn]
  }
}

resource "aws_iam_role_policy" "ssm_session_kms" {
  name   = "ssm-session-kms"
  role   = aws_iam_role.app_ec2.id
  policy = data.aws_iam_policy_document.ssm_session_kms.json
}

The managed policy AmazonSSMManagedInstanceCore does not include KMS permissions for your own keys, so this has to be added explicitly for every instance role that should accept sessions.

Step 2: keep the key policy explicit

The default key policy delegated to IAM, which worked. For an audited key we preferred an explicit statement as well, so the key policy alone documents who may use it:

{
  "Sid": "AllowSessionUseByInstanceRoles",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/app-ec2-role" },
  "Action": "kms:Decrypt",
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:CallerAccount": "111122223333" }
  }
}

A matching statement allows the engineers' SSO permission set roles to call kms:GenerateDataKey.

Step 3: give private subnets a path to KMS

Instances in subnets without a NAT gateway reach AWS APIs only through interface VPC endpoints. We had endpoints for SSM but not for KMS:

resource "aws_vpc_endpoint" "kms" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.eu-west-1.kms"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private_isolated[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

The endpoint security group allows HTTPS from the VPC CIDR. With private DNS enabled, the agent's calls to kms.eu-west-1.amazonaws.com resolve to the endpoint without any agent configuration.

Step 4: roll out in the right order

The real mistake was the order of changes. The safe sequence is:

1. Create the key and its policy.
2. Add kms:Decrypt to every instance role and kms:GenerateDataKey to user roles.
3. Add KMS VPC endpoints where needed.
4. Test a session on one instance in each network type with a test preference document.
5. Only then switch the default Session Manager preferences to use the key.

Alternatives and trade-offs

• Roll back the preference. We did this first to restore access in five minutes, then applied the fix and re-enabled it.
• Keep the break-glass path. We added EC2 Serial Console access for two admin roles, which saved us here because it does not depend on Session Manager.

Confirming in production

aws ssm start-session --target i-0abc123def4567890
# Starting session with SessionId: alice@example.com-...
# sh-5.2$

The agent log showed the handshake completing:

INFO [SessionManager] ... Session Manager encryption enabled, handshake completed successfully

CloudTrail recorded Decrypt calls on the key from the instance role for each session, and GenerateDataKey from the engineer's role, which also gave the audit team the evidence they wanted.

Regression guard

• A Terraform module for instance roles now includes the KMS statement by default, so new roles cannot miss it.
• A synthetic check opens a session to one instance per subnet type every hour and pages on failure.
• AWS Config tracks the Session Manager preferences document, alerting on changes outside the infrastructure pipeline.

Security implications

The change improves confidentiality of session data at rest in logs and adds KMS-level audit records of every session. Granting kms:Decrypt to instance roles is narrow when scoped to this one key. Do not use a wildcard resource, or instances could decrypt other data. The outage itself was an availability problem for operations access; keeping a break-glass path that does not depend on the same feature is part of the fix.

Prevention checklist

• Before enabling Session Manager KMS encryption, grant kms:Decrypt on the key to every instance role.
• Add a KMS interface endpoint to any subnet without internet egress.
• Test with one instance per network type before changing default preferences.
• Keep a break-glass access path independent of Session Manager.
• Monitor sessions with a synthetic check, not only agent online status.

References

• AWS Systems Manager, Enable KMS key encryption of session data: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-enable-encryption.html
• AWS KMS, Key policies: https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html
• AWS Systems Manager, VPC endpoints: https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html
• AWS KMS, Connecting through a VPC endpoint: https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html