Google Cloud’s London outage: why one zone affected the region

How did a single-zone failure affect the region?

This was not an event limited to resources in europe-west2-b. A UK data center facility hosted that zone. After it lost power, regional services had to handle redirected queries, high-availability database failovers and compute workloads. Google's report says the extra demand consumed compute slots and processing capacity in neighboring zones. Queues, delayed database failovers and intermittent 500 and 503 errors followed across a broader set of europe-west2 services.

The report confines the physical power failure to europe-west2-b, while also describing effects on zones that did not lose power and on regional architectures. It attributes those wider effects mainly to demand surges and dependencies across zones in the control plane, the systems that manage resources and schedule work. Google said it identified no impact outside europe-west2. An error elsewhere in a region does not mean every facility there lost power.

Why did the redundant feeds and UPS not contain the outage?

The facility normally had independent A and B utility feeds, a 2N arrangement intended to provide duplicate power paths for critical loads. Before the incident, however, the B feeder breaker had been left open for repairs after an issue found during an Oct. 14 generator load test. The site was temporarily in a 1N maintenance posture, with feed A carrying the critical load. The report says uninterruptible power supply (UPS) batteries could sustain that load for about five minutes, bridging the time needed to detect undervoltage, start and synchronize generators, and transfer the load.

At 22:53 US/Pacific on Oct. 19 (05:53 UTC on Oct. 20), an external underground cable fault on the utility network interrupted feed A. The programmable logic controller (PLC) voltage sensors were upstream of the feeder breakers, and the logic did not check both undervoltage and breaker position. Although B was isolated and had no downstream power, it remained energized upstream. Both master PLCs used the same logic and treated that voltage as a healthy source, so they did not signal the generator plant to start. The UPS reserves ran out and the data halls lost power. Duplicate feeds and controllers could not compensate for a temporary single-feed state and a shared sensing and logic flaw.

What does the status feed's six-hour window measure?

The final incident report says customer impact began at 23:00 US/Pacific on Oct. 19 and that most workloads had recovered by 05:08 on Oct. 20. That is 6 hours 8 minutes, or 06:00-12:08 UTC on Oct. 20, which matches the saved status feed interval. An earlier status update gave 05:10 and 6 hours 10 minutes; the later report supersedes those figures. Do not combine the two versions.

The report gives different recovery milestones. At 04:11 US/Pacific, Google Cloud services and APIs were available to customers, although some Compute Engine VMs were still unavailable. At 04:29, regional services and backend compute management were fully returned to handling traffic, ending the core platform-level disruption. At 05:08, most workloads had recovered; 99.5% of Compute Engine VMs were operational and healthy, and new VM creation was supported. The feed's 06:00-12:08 UTC interval is the main incident window, not a universal duration for every product, disk or instance.

Which products had their own recovery tails?

All product times in this section use the report’s US/Pacific time zone (UTC−7 on these dates). Times starting with 23 refer to Oct. 19; the early-morning and daytime times refer to Oct. 20. BigQuery reported a regional disruption from 23:03 to 04:12, about 5 hours 9 minutes, with query queues, delayed jobs, intermittent 503 errors on job APIs and some job failures.

GKE’s report gives a 23:02 start, initial mitigation at 05:22 and final resolution at 06:37, with a stated total duration of 7 hours 34 minutes. Zonal clusters in europe-west2-b were unavailable from 23:05 to 04:30 and fully restored by 06:00. The 04:30 time is a product recovery milestone; data hall power had returned earlier, at 00:20. These are separate events.

Cloud SQL automatically started failovers for regional high-availability instances, but some recovery depended on Regional Persistent Disk. Most of the remaining instances recovered by 01:00 on Oct. 20; a small number did not fully recover until 11:00. AlloyDB reported more than 99% of affected instances healthy by 06:16, with the last zonal primary instances recovering by 08:28. Apigee X's runtime returned elevated HTTP 5xx errors for 7 hours 19 minutes. The report says two of its three datastore replicas had been placed in the failed zone, leaving the datastore without quorum, the majority of replicas needed to keep operating. These are product-specific findings, not evidence about the state of any other instance.

Does an available incident record mean every resource is back?

In the feed saved for this review, the incident entry was marked AVAILABLE and its current affected-locations list was empty. That describes the state of the incident entry at capture time. The report still documents different recovery times for VMs, databases, clusters and APIs. A closed regional notice or a restored API does not, by itself, prove that a particular project, instance or disk is healthy.

The report records PLC reconfiguration and validation as complete on Oct. 26. It estimates repair of primary feed A by Nov. 13, and transfer back to feeds A and B with coordinated testing by Nov. 17. A comprehensive PLC code review has an estimated completion date of Nov. 6 and is required before the generator transfer test. These are the statuses in the report modified Nov. 5. Checking the same report on Nov. 6 did not establish that the estimated work had finished; reaching a date is not confirmation of completion.

What should operators compare when checking their own impact?

Keep the time and time zone for affected requests or jobs, along with the product, region, zone and resource type. Compare those timestamps with the product-specific sections of the incident record instead of relying only on the 6-hour-8-minute headline or the status badge. In this incident, API availability, restoration of regional backend traffic and recovery of individual products did not happen at the same time.

Then check the normal product console for the state of the instance, disk, cluster or database. If an individual resource remains unhealthy, use those timestamps and its resource identifier in the established support process. The incident report explains the published facility and service timelines; it does not establish the final result of each customer's request, data or resource.