When we finally split our bill by environment, development, staging and preview environments were 38 percent of total cloud spend. Most of it ran 24 hours a day for people who work 8. This guide covers schedules, ephemeral environments, right-sizing, data copies and guardrails.
A partial outage in one AWS availability zone took down our search API completely, although it ran three replicas on a three-zone cluster. The scheduler had placed all three in the same zone. How the default spreading works, why it was not enough, and the topologySpreadConstraints we now require.
We added a new GPU node pool and training pods stayed Pending with Insufficient nvidia.com/gpu, while the nodes sat idle. The NVIDIA device plugin was not running on the new nodes because its DaemonSet did not tolerate their taint, so the nodes never advertised GPUs.