Every second build on our Jenkins agent failed at checkout with Permission denied while deleting node_modules. A docker run step in the pipeline had written files as root into the bind-mounted workspace. How we proved it, fixed the user mapping and made cleanup reliable.
Our first Site-to-Site VPN between an office firewall and AWS came up green on both sides, yet pings failed and, once they worked, large file transfers hung. Three separate problems: route propagation, the firewall's traffic selectors and MTU. How we found each one.
One Terraform state for a whole environment meant 11-minute plans, fragile applies and an app deploy that could touch the VPC. How we split it into layers by rate of change and blast radius, passed values between them, moved resources with removed and import blocks, and kept the apply order sane.
A Node.js API image had grown to 1.1 GB, slowing every deploy and autoscaling event. Step by step, we measured where the size came from, moved to a multi-stage build, chose a base image deliberately and cleaned the build context, ending at 96 MB, with the trade-offs of each step.
We inherited an AWS account built by hand: 60 resources nobody dared to touch. Terraform import blocks with -generate-config-out let us adopt them in reviewed pull requests with a zero-change plan as proof. The workflow, the cleanup the generated code needs, and the traps.
Terraform's S3 backend can lock state with a lock file in the bucket itself, and DynamoDB-based locking is deprecated. How S3 native locking works, the IAM it needs, a safe migration with both mechanisms on during the transition, and the failure modes we tested.
Why does deleting a file in a later Dockerfile step not shrink the image? Why is the first write to a large file inside a container slow? A hands-on explanation of image layers and overlayfs, built with mount commands you can run yourself, and what it means for images and databases.
After marking a variable sensitive to keep database passwords out of plan output, terraform plan failed with Invalid for_each argument. Sensitive values cannot be for_each keys because keys appear in resource addresses. How we restructured the input so keys stay public and secrets stay hidden.
Our analytics VPC could reach the shared services VPC, and shared services could reach the payments VPC, but analytics timed out connecting to payments. VPC peering does not route traffic through a middle VPC. How we proved it, the options, and why we moved to Transit Gateway.
Our shared Terraform modules broke consumers twice in one quarter. We added terraform test: fast plan-mode tests with mocked providers on every pull request, and a nightly apply-mode suite against a sandbox account. The test layout, what to assert, and what it costs.