Our GitHub Actions pipeline for a serverless API failed every time a merge did not change the stack, such as documentation-only commits: Error: No changes to deploy. How CloudFormation empty changesets work, the flag that makes sam deploy exit cleanly, and the samconfig and OIDC setup we use per environment.
We moved CI to self-hosted GitHub Actions runners on EKS with Actions Runner Controller. Every job that built an image failed: Cannot connect to the Docker daemon. Runner pods have no Docker by default. How dind and kubernetes container modes work, what each costs in security, and the node pool we gave them.
Our main pipeline took 34 minutes, and engineers batched changes to avoid waiting, which made reviews worse. We cut it to 14 minutes without new hardware magic: measure the critical path, parallelise, cache the right layers, split tests by timing and build only what changed.
Our versioning workflow created a tag and pushed it, and the release workflow that listens for tags never started. Events created with the default GITHUB_TOKEN do not trigger new workflow runs. We switched to a GitHub App token for the push and made the release explicit with workflow_call.
A lockfile-only dependency bump passed CI and then broke production. Our cache key hashed package.json, not package-lock.json, so CI restored old node_modules and skipped npm ci. How we found it and rebuilt dependency caching we can trust.
We enabled BuildKit's GitHub Actions cache and our multi-stage Go build still took 9 minutes on every run. The default cache export mode only stores layers of the final image, not the builder stage where all the time is spent. mode=max and per-branch scopes cut it to 2 minutes.
An internal audit found a workflow that used pull_request_target to post preview links and also checked out the pull request's code and ran npm install. Any fork PR could have run code with our deploy token. We split it into an unprivileged build and a privileged comment step.
In March 2025 the popular tj-actions/changed-files action was compromised and its version tags were repointed to code that printed CI secrets into workflow logs. Three of our repositories used it by tag. This is how we checked exposure, rotated secrets and moved to SHA pinning.
We added five scanners to CI and within a week developers were merging with admin override. This is the gate design we rebuilt: block only new, fixable, high-signal findings on PRs, scan everything nightly, and make exceptions expire.
After a provider upgrade done on a MacBook, terraform init failed on Linux CI runners with a checksum mismatch against the lock file. The lock file only had hashes for darwin_arm64 packages. Locking all platforms we use fixed it without weakening verification.
Every deploy of one service failed with another operation is in progress after someone cancelled a pipeline mid-upgrade. The release was stuck in pending-upgrade. We recovered with helm rollback, then added CI concurrency, timeouts and atomic upgrades so it cannot recur.
After moving repositories into a new GitHub organization, our coverage-comment and release-tag steps failed with 403 Resource not accessible by integration. The GITHUB_TOKEN defaulted to read-only. A scoped permissions block fixed it, and fork PRs needed a different design.
Our deploy job worked on main and failed on the production environment with Not authorized to perform sts:AssumeRoleWithWebIdentity. Using a GitHub environment changes the OIDC subject claim. We rewrote the trust policy around exact subjects instead of wildcards.