At 03:03 someone edits a ConfigMap. By 03:12 you have a dozen alerts, and none of them mentions the edit. This post shows why Kubernetes alerts pile up and how to cut them down with Prometheus and Alertmanager.
One change, many symptoms
A teammate edits a ConfigMap at 03:03 and removes a key the app needs. Running pods keep the old values, so nothing breaks yet. At 03:04 a rollout of checkout-api starts new pods, and they read the new file and cannot boot. The cluster reports this over the next three minutes.
03:03 ConfigMap checkout-config edited 03:04 Deployment checkout-api rollout started 03:04 Pod checkout-api-7f9c CrashLoopBackOff 03:04 Pod checkout-api-7f9c readiness probe failed 03:05 Deployment checkout-api 0 of 3 replicas available 03:05 Service checkout-api no ready endpoints 03:06 HPA checkout-api unable to scale 03:06 Event Back-off restarting failed container (x14)
Enter fullscreen mode
Exit fullscreen mode
Each line can fire its own alert.…