Most Prometheus setups I see have alerts but no answer to the 3 a.m. question: "OK, it fired. Now what?" The fix is boring and effective: every alert gets a runbook_url annotation that points to one short page with the same structure every time.
I wrote such a runbook for 44 common alerts (Linux hosts via node-exporter, Docker via cAdvisor, Kubernetes via kube-state-metrics, websites and TLS via blackbox-exporter) and put it online for free:
π Prometheus alert runbook (free, English)
The format
Every section answers the same five questions:
What it means : in plain words, including the exact condition.
Common causes : the usual suspects, most likely first.
Diagnostics : copy-paste commands.
How to fix .
When it is OK to silence : so silences get a reason and an expiry instead of becoming "monitoring switched off".
Example: disk will fill within 24 hours
A static "disk > 90%" alert is either too late or too noisy.β¦