#scheduler

DevOps & CI/CD Bug Report

Kubernetes Pods stuck Pending: Insufficient cpu, untolerated taints and node affinity

John Nguyen · Jul 20, 2025 · 9 min read

A routine rollout left 14 Pods Pending with Insufficient cpu while dashboards showed nodes at 30 percent CPU. The scheduler counts requests, not usage, and a new tainted node pool hid the free capacity. Here is how we read the FailedScheduling message and fixed it.

DevOps & CI/CD

From kubectl apply to a running container: the Kubernetes Pod lifecycle step by step

John Nguyen · Nov 5, 2024 · 9 min read

kubectl apply returns in milliseconds, but a container can take a minute to start. We trace every hop: API server, admission, etcd, controllers, scheduler, kubelet, CRI, CNI and probes, and show which command reveals each stage.

Incidents & Postmortems Bug Report

Nightly billing job ran twice on the DST fall-back night and double-charged customers

Daniel Tran · Nov 4, 2024 · 9 min read

On November 3, 2024, our 01:30 billing run fired at 01:30 CDT and again at 01:30 CST, charging 1,284 subscriptions twice. The in-house scheduler matched local wall-clock time. We moved schedules to UTC and added idempotency keys, a run ledger and a unique constraint.