November 2026 | ~18 min read
The Shared Cluster That Wasn't Sharing
Three incidents in one month. Same cluster. Different root cause every time. All preventable.
Week 1: Noisy neighbor. The data team kicked off a backfill job — 40 pods, no resource limits, no dedicated node pool. Within minutes, the job consumed 80% of cluster CPU. The payments API started returning 504s. On-call spent two hours tracking it down because kubectl top nodes showed everything was "healthy" — the nodes were just saturated. Revenue impact: $12,000 in failed transactions.
Week 2: Accidental scraper. A misconfigured frontend pod had a Prometheus scrape target pointing at * instead of its own namespace. It was hitting every metrics endpoint in the cluster every 15 seconds. Prometheus's memory ballooned from 4GB to 28GB, OOMKilled, and took every Grafana dashboard with it. The monitoring team didn't notice for 3 hours because… their monitoring was down.
Week 3: Wrong namespace delete.…