Since the first week of this project, observability/prometheus/alerts.yml has contained this:
- alert : PDSCircuitOpen expr : pybreaker_state{name="pds"} == 1
Enter fullscreen mode
Exit fullscreen mode
It is a correct rule. The circuit breaker in front of the Personal Demographics Service opens after five consecutive failures, the pipeline carries on with the demographics from the message, and somebody should be told.
No service has ever exported pybreaker_state . The rule has been watching a series that does not exist, which looks exactly like a healthy system.
The problem
Monitoring has a particular failure mode: you cannot tell the difference between nothing being wrong and nothing being watched. A dashboard with no red on it means one of those two things, and it does not say which.
In this pipeline, that matters more than usual, because the failures I care about mostly return 200. A patient written without an NHS number to match on is a successful request.…