Pearl · CR-E-020

ATS Mission Path and Failover Capacity

Prove that each ATS can transfer the actual critical load onto a correctly rated and protected failover path without an unintended trip or loss of redundancy.

Construction terms explained

ATS:: automatic transfer switch
IST:: integrated systems testing
MOP:: method of procedure

Critical power Generators

What to check before the work begins

Before this representative loaded ATS transfer proceeds, has the project reconciled the actual normal and failover load, the capacity of every consequential path, and the complete installed trip settings against the current approved design, then demonstrated the transfer and retransfer under representative load?

Reconcile current normal load and maximum credible coincident failover load for each affected ATS path
Verify source, conductor, ATS, breaker, and downstream path ratings against that load
Reconcile every installed ATS and upstream/downstream protective setting against the final approved protection and coordination basis
Confirm the approved sequence of operations, transfer permissives, reset behavior, and restoration route
Execute an approved representative loaded transfer and retransfer with synchronized event capture
Disposition every unexpected trip, lockout, reset, overload, or manual intervention through the existing commissioning issue process

Use in the project example

What the sources describe

On May 8, 2024, a public-utility issue caused an undervoltage condition followed by power loss at a third-party Google Cloud data center in Sydney. Operating current exceeded the trip settings of both paired ATS units feeding affected rows. Google's investigation found that the ATS trip settings were not in accordance with the site design. On-site engineers manually closed breakers to restore power, and affected services experienced impact lasting up to 2 hours and 55 minutes.

Evidence to confirm

Current normal and maximum credible coincident failover load by ATS path
ats or switchgear oem · Before the relevant work begins
Loads are revision-controlled, reflect the planned test condition, and are accepted by the commissioning authority and responsible electrical engineer.
Source, conductor, ATS, breaker, and downstream path rating reconciliation
ats or switchgear oem · Before the relevant work begins
Every consequential component is cross-referenced to the approved failover load and no unresolved capacity exception remains.
Field-verified ATS and protective-device settings register
ats or switchgear oem · Before the relevant work begins
Installed settings match the final approved protection and coordination basis or carry an approved exception.
Approved loaded-transfer MOP with expected results, abort criteria, and safe restoration
ats or switchgear oem · Before the relevant work begins
The MOP identifies the representative load, source states, roles, hold points, and return-to-normal sequence.
Synchronized ATS, breaker, current, voltage, and event records for transfer and retransfer
ats or switchgear oem · Before the relevant work begins
The approved sequence completes without unexplained trip, overload, lockout, loss of required redundancy, or undocumented manual recovery.
Disposition of every unexpected trip, lockout, reset, overload, or manual intervention
ats or switchgear oem · Before the relevant work begins
All consequential anomalies are closed, retested successfully, or retained under formal owner risk acceptance.

Conditions to resolve before proceeding

Installed settings do not match the approved design or protection basis
A failover path lacks demonstrated capacity for the approved coincident load
The current load or topology cannot be reconciled against the transfer-test basis
An unexplained trip, lockout, overload, or manual recovery occurs

When this applies

Integrated systems testing
Functional performance testing
Installation turnover

Assess each lesson against your project, equipment, and local requirements with the people responsible for the work.

Limitations and context

Applies when: Paired or redundant ATS paths supply consequential critical load
Applies when: A representative loaded transfer or retransfer is planned
Applies when: Trip settings and failover loading can be reconciled against the final site design
Outside scope: Projects with no ATS or equivalent automatic source-transfer device
Outside scope: Transfer paths whose interruption has no consequential project or operating effect
Outside scope: Configurations whose applicability has not been confirmed by the commissioning authority
The incident occurred at an unnamed third-party data center, limiting project-specific comparison.
The public incident report does not publish the ATS model, numerical settings, actual row load, coordination study, one-line diagram, or complete breaker-event record.
The report does not establish whether a particular acceptance or integrated-systems test was omitted before the event.
The report does not identify the third-party facility owner or publish the commissioning and maintenance records.
The report does not publish the ATS manufacturer or model, numerical trip settings, actual affected-row load, or the complete coordination study.
The source verifies the settings discrepancy and operational consequence but does not prove that a particular commissioning test had been omitted.
A loaded transfer can be performed or represented safely under an approved method.
The project team can obtain final field settings and a current load/failover matrix.
The commissioning authority will route the check through existing MOP and IST governance rather than a separate Build Pearls approval.

Where the lesson comes from

Sources

Use the original material to understand the evidence, scope, and context behind this Pearl. Suggested project actions are Build Pearls’ interpretation.

Multiple services impacted in australia-southeast1 — Incident Report
Google Cloud · Source date: 2024-05-21 · Incident Report — Remediation and Prevention, action list; Incident Report — Remediation and Prevention, first paragraph; Incident Report — Root Cause, first and second paragraphs; Incident Report — Root Cause, first paragraph; Incident Report — Root Cause, second paragraph; Incident Report — Summary · Accessed: 2026-08-08
Open source