Menu

The sensor that said the power was fine while a whole cloud zone went dark
📰

The sensor that said the power was fine while a whole cloud zone went dark

DEV Community: go·Kashif Manzer·9 days ago
#Q76oHj4k
#devtogo#devto#dev#zone#power
Reading 0:00
15s threshold

At 06:53 on Tuesday 20 October 2026, an underground power cable failed outside a data centre in London. The building lost one of its two power feeds. That should have been a non-event. The site had a second feed, a bank of diesel generators, and batteries to cover the gap while the generators started.

Six hours later, an entire Google Cloud zone was still dark. Forty services were affected. Banks could not process payments properly. And the generators, which had fuel and were ready to run, never received the order to start.

The reason is the kind of bug that should make every backend developer uncomfortable, because most of us have written its smaller cousin. A sensor was measuring the wrong thing, and the system that trusted it had no way to know.

How the site was supposed to work

The facility hosts europe-west2-b, one of three availability zones in Google Cloud's London region.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.