CI pipeline red on "No changes to deploy": sam deploy, empty changesets and environment config

We deploy a serverless API with AWS SAM from GitHub Actions: build, then sam deploy to staging on every merge to main, and to production on tags. One day a pull request that only changed the README was merged, and the pipeline failed:

Initiating deployment
=====================

Waiting for changeset to be created..

Error: No changes to deploy. Stack orders-api-staging is up to date
Error: Process completed with exit code 1.

From then on, any merge that did not change the stack template or function code turned the pipeline red: documentation updates, test-only changes, workflow edits. The team started ignoring red builds on main, which is worse than the original problem.

What sam deploy does

sam deploy uploads artifacts, then creates a CloudFormation changeset describing the difference between the deployed stack and the new template. If the changeset is empty, CloudFormation marks it FAILED with the reason "The submitted information didn't contain changes", and SAM reports "No changes to deploy" and exits with status 1.

aws cloudformation describe-change-set --stack-name orders-api-staging --change-set-name samcli-deploy1759...

"Status": "FAILED",
"StatusReason": "The submitted information didn't contain changes. Submit different information to create a change set."

Why "nothing changed" when we rebuilt

SAM builds are deterministic for our Python functions: the same source and dependencies produce the same zip, with the same hash, so the S3 artifact key and therefore the template stayed identical. That is the desired behaviour. It just collides with a deploy step that treats "no difference" as an error.

What we ruled out

• Stale credentials or a wrong account. The changeset was created in the right stack; it was empty.
• Changes hidden by caching. A real code change produced a non-empty changeset as expected.
• Drift. CloudFormation changesets compare templates and parameters, not live resources, so drift is irrelevant here.

Root cause

sam deploy, like aws cloudformation deploy, exits with a non-zero status when the changeset is empty, unless told otherwise. Our pipeline did not handle that case, so every merge without infrastructure or code changes failed the deploy job.

Empty changesets are a normal outcome

In a pipeline that deploys on every merge, most merges in a mature service do not change the deployed stack. Treating that as an error is a choice the tools leave to you. Both sam deploy and aws cloudformation deploy have a flag for it:

--no-fail-on-empty-changeset    Return exit code 0 when there are no changes to be made to the stack.

Where configuration should live

The other half of our problem was that pipeline steps had grown long command lines with stack names, regions, capabilities and parameter overrides duplicated per environment. SAM's samconfig.toml supports named environments, which keeps that configuration in the repository, next to the template, and reviewed like any other code. The pipeline then only has to choose the environment name.

Who is affected

Every pipeline that runs sam deploy or aws cloudformation deploy on each merge, including multi-stack pipelines where most stacks are unchanged on a given commit. Teams that use --no-execute-changeset to review changesets see the same behaviour: the empty changeset fails to create. Pipelines that run cdk deploy are not affected in the same way, because the CDK CLI skips unchanged stacks by default.

Environment

samconfig.toml with environments

version = 0.1

[default.global.parameters]
stack_name = "orders-api"

[default.build.parameters]
cached = true
parallel = true

[staging.deploy.parameters]
stack_name = "orders-api-staging"
region = "eu-west-1"
s3_prefix = "orders-api-staging"
resolve_s3 = true
capabilities = "CAPABILITY_IAM"
confirm_changeset = false
fail_on_empty_changeset = false
parameter_overrides = "Stage=staging LogLevel=info"
tags = "service=orders-api environment=staging"

[prod.deploy.parameters]
stack_name = "orders-api-prod"
region = "eu-west-1"
s3_prefix = "orders-api-prod"
resolve_s3 = true
capabilities = "CAPABILITY_IAM"
confirm_changeset = false
fail_on_empty_changeset = false
parameter_overrides = "Stage=prod LogLevel=warn"
tags = "service=orders-api environment=prod"

fail_on_empty_changeset = false is the config file form of --no-fail-on-empty-changeset.

The workflow

name: deploy
on:
  push:
    branches: [main]
    tags: ["v*"]

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ startsWith(github.ref, 'refs/tags/') && 'prod' || 'staging' }}
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/setup-sam@v2
        with:
          use-installer: true
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.DEPLOY_ROLE_ARN }}
          aws-region: eu-west-1
      - run: sam build
      - run: sam deploy --config-env ${{ startsWith(github.ref, 'refs/tags/') && 'prod' || 'staging' }}

Each GitHub environment has its own DEPLOY_ROLE_ARN variable, and the IAM role trust policy only allows that repository and environment:

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:example-org/orders-api:environment:prod"
  }
}

Skip the deploy when nothing deployable changed

Exiting cleanly on an empty changeset fixed the red builds, but each run still spent two minutes building and creating a changeset. For pushes to main, we added a path filter so documentation-only changes do not start the deploy job at all:

on:
  push:
    branches: [main]
    paths-ignore: ["**.md", "docs/**", ".github/ISSUE_TEMPLATE/**"]

Tags always deploy.

Alternatives and trade-offs

• Catching the error in shell (sam deploy || grep "No changes to deploy"). Works, but string matching on error messages breaks when messages change. The flag is the supported way.
• Deploying with sam sync. Faster for development, not intended for production pipelines.

What about stack outputs?

One consequence of skipping a deploy: steps that read stack outputs after sam deploy, such as the API URL for smoke tests, must not depend on the deploy having happened in this run. Ours now read outputs with aws cloudformation describe-stacks after the deploy step, which works whether the stack changed or not, and the smoke tests run on every merge.

How to test the fix

Push a README-only commit to a test branch with the workflow pointed at a sandbox stack, and confirm the deploy step logs "No changes to deploy" and exits 0:

sam deploy --config-env staging; echo "exit=$?"
# Error: No changes to deploy. Stack orders-api-staging is up to date
# exit=0

Confirming in production

Over the following month, the main pipeline stayed green for 46 merges, 19 of which produced empty changesets. Production deploys from tags were unaffected.

Regression guard

• Deploy configuration lives in samconfig.toml, reviewed with code; workflow files only select the environment.
• A required status check on main ensures red builds block merges, which only works because red now means something.

Security implications

Moving to OIDC with per-environment roles removed long-lived AWS access keys from GitHub secrets. The sub condition pins each role to one repository and one GitHub environment, so a workflow in another repository, or a pull request job without the environment, cannot assume the production role. Production deploys additionally require approval through GitHub environment protection rules.

Prevention checklist

• Treat empty changesets as success in deployment pipelines.
• Keep per-environment SAM settings in samconfig.toml, not in long command lines.
• Use OIDC roles scoped to repository and environment instead of access keys.
• Skip deploy jobs for changes that cannot affect the stack.
• Keep pipelines green so red builds are noticed.

References

• AWS SAM CLI, sam deploy: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html
• AWS SAM CLI configuration file: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
• Configuring OpenID Connect in Amazon Web Services: https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
• CloudFormation change sets: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets.html