New Pods failed with CreateContainerConfigError while old ones kept running. A key renamed in the cloud secret store made External Secrets rewrite the Kubernetes Secret without DATABASE_URL. We show how to find the missing key fast and how to fail before rollout next time.
A security review found that our password reset tokens and temporary passwords came from Python's random module. Its Mersenne Twister generator is fast and statistically good, but not unpredictable. We moved to secrets, hashed tokens at rest, and added Bandit to CI.
GitOps says everything lives in Git, and secrets are the exception that breaks the rule. This guide compares encrypting secrets in Git with SOPS or Sealed Secrets against referencing them from a secret manager with External Secrets Operator or the CSI driver, including rotation and bootstrap.
After marking a variable sensitive to keep database passwords out of plan output, terraform plan failed with Invalid for_each argument. Sensitive values cannot be for_each keys because keys appear in resource addresses. How we restructured the input so keys stay public and secrets stay hidden.
An internal audit found a workflow that used pull_request_target to post preview links and also checked out the pull request's code and ran npm install. Any fork PR could have run code with our deploy token. We split it into an unprivileged build and a privileged comment step.
In March 2025 the popular tj-actions/changed-files action was compromised and its version tags were repointed to code that printed CI secrets into workflow logs. Three of our repositories used it by tag. This is how we checked exposure, rotated secrets and moved to SHA pinning.
A developer committed a .env file with a live AWS access key, then deleted it in the next commit. The key stayed in history. We revoked it within 40 minutes, checked CloudTrail, rewrote history with git filter-repo and added scanning so it cannot happen again.