Menu

1 / 2

Part 4: The Alert That Could Never Fire

#devcommunity#dev#breaker#number#test
Reading 0:00
15s threshold

Since the first week of this project, observability/prometheus/alerts.yml has contained this:

- alert : PDSCircuitOpen expr : pybreaker_state{name="pds"} == 1

Enter fullscreen mode

Exit fullscreen mode

It is a correct rule. The circuit breaker in front of the Personal Demographics Service opens after five consecutive failures, the pipeline carries on with the demographics from the message, and somebody should be told.

No service has ever exported pybreaker_state . The rule has been watching a series that does not exist, which looks exactly like a healthy system.

The problem

Monitoring has a particular failure mode: you cannot tell the difference between nothing being wrong and nothing being watched. A dashboard with no red on it means one of those two things, and it does not say which.

In this pipeline, that matters more than usual, because the failures I care about mostly return 200. A patient written without an NHS number to match on is a successful request.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.