# [[2025-06-12 GCP Outage]] _Created: 2025-06-15_ | #google #gcp | [[Google]] On June 12 2025, there was massive worldwide outage in GCP. No one in my office could log into Google Meet, GCP Console was reporting that no resources could be found (causing minor panic), and our apps depending on Gemini were all timing out. Google posted a [postmortem report](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW). Summarizing the post mortem here for my memory. ## Timeline **May 29:** A code change to the quota policy system was deployed worldwide, gradually region by region. However, the feature to be tested was not exercised because it needed a suitable policy to exercise it, and such a policy was not added alongside. >[!Important] Guardrails on the new feature > The new feature had a red button configuration which on deployment could disable the new code path. However, *it wasn't behind a feature flag*. **June 12:** A policy exercising the new code feature was added. Policies are dynamic and global, causing it to be applied everywhere within minutes. A bug in the previously deployed code in response to the policy caused the quota policy application to terminate due to a *null pointer exception*. Within 40 minutes, SREs detected, and implemented the red button config change everywhere, causing recovery to start. **However**, on large regions like `us-central1`, restarting of the policy engine caused a *herd effect* on the underlying *Spanner* table causing infrastructure overload. Engineers manually throttled restarts to mitigate but it took ~3 hours for things to stabilize. Unfortunately, most of my company infrastructure was in `us-central1` and was badly impacted. >[!info] Delay in status notification >There was a lot of anger about why Google took so long to report the problem in their status web pages. Turns out their *Cloud Service Health infrastructure* itself was impacted by the outage. 🤦‍♂️ ## Other analysis of the outage - [ByteByteGo: How the Google Cloud Outage Crashed the Internet](https://blog.bytebytego.com/p/how-the-google-cloud-outage-crashed)