We moved private course videos behind CloudFront signed cookies. The app set the three cookies, yet the browser got 403 Missing Key-Pair-Id for every segment: cookies were scoped to the app's host, not the CDN's, and the behaviour trusted a different key group. How we debugged it with curl.
A DuckDB report job queried Parquet files on S3 fine on a laptop and failed with HTTP 403 in its ECS task. Locally DuckDB used a persistent secret from personal credentials; in the container nothing told it to use the task role. How DuckDB secrets and the credential chain work, plus the KMS detail.
A researcher reported that CSV exports with customer emails were downloadable from an S3 bucket. The cause was a bucket policy granting public read, added years earlier for a static site. How we contained it, proved who accessed what, and locked down every bucket.
A refactor changed one output prefix in an image processing Lambda, and the function started triggering itself. Invocations went from 40,000 a day to 9 million in six hours. We stopped it with reserved concurrency, then separated input and output and added loop guards.
Terraform's S3 backend can lock state with a lock file in the bucket itself, and DynamoDB-based locking is deprecated. How S3 native locking works, the IAM it needs, a safe migration with both mechanisms on during the transition, and the failure modes we tested.
After we redeployed our MLflow tracking server on Kubernetes, every run older than the deploy showed metrics but no artifacts, and model loading failed. The server wrote artifacts to a local directory in the pod. We moved to proxied artifact storage on S3 and recovered what we could.
Our AWS bill jumped by over 9,000 USD in one month and nothing looked broken. The cause was NAT Gateway data processing: batch jobs in private subnets were reading S3 and pulling ECR images through NAT. VPC endpoints cut that line item by 94 percent.
A Lambda function that converts uploaded videos to thumbnails worked for the first few invocations, then failed with No space left on device. Files written to /tmp survived across warm invocations of the same execution environment and filled the 512 MB default. How we found it and fixed cleanup, sizing and streaming.
Receipt uploads to S3 failed with 403 SignatureDoesNotMatch for about 4 percent of users, mostly iPhone photos uploaded from Windows. The URL was signed with one Content-Type and the browser sent another. Reading the CanonicalRequest in the error pinpointed it.
A cancelled GitLab pipeline killed terraform apply mid-run and left the S3 backend locked. Every plan after that failed with Error acquiring the state lock. How we proved no run was active, unlocked safely, checked for drift, and stopped CI from cancelling applies.
Our order fulfilment state machine ran for months, then failed for large orders with States.DataLimitExceeded. A Map state collected per-item results into one array that crossed the 256 KiB payload limit. How we found the state that grew, trimmed outputs with ResultSelector and moved bulk data to S3.
Our app said the media bucket held 4 TB, while S3 billed us for 31 TB. Versioning kept every overwritten and deleted object as a noncurrent version, and failed multipart uploads left invisible parts behind. Lifecycle rules brought billed storage down to 5.2 TB.