off-prem

October outage caused by bad PLC config exposed at a time maintenance work reduced resilience

Google Cloud has revealed that the cause of the October 20 outage in its europe-west-2-b availability zone was a ‘misconfiguration’ of its systems that meant its emergency generators didn’t start.

The outage at the London facility caused service interruptions at outfits including Lloyds, HSBC, Royal Mail, and Revolut .

In a November 4 update to its service status page, Google explained what went wrong.

“The site relies on a 2N utility power architecture — where the facility has complete duplicate power paths for every critical load — backed by a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller (PLC),” Google revealed.

PLC misconfiguration prevented the facility’s generator plant from receiving an automatic start signal

The facility gets its juice from two independent utility feeders, which Google calls “A” and “B”.…