On July 2nd, a Sunday, the unthinkable happened. The technical experts at Google misconfigured their systems resulting in a major outage for two and a half hours. As a result, juggernaut Google properties such as YouTube and Google cloud suffered a huge loss in traffic. In that critical amount of time, YouTube traffic dropped 10% and Google Cloud Storage traffic dropped 30%. Moreover, the mishap negatively affected one percent of more than one billion Gmail users. The big name customers that were affected include Snapchat, Vimeo, Shopify, Discord, and Pokemon GO.
Google thoroughly investigated the issue and offered to the public an apology and explanation as well as a technical report. In the report, they state, "Customers may have experienced increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1. Google Cloud instances in us-west1, and all European regions and Asian regions, did not experience regional network congestion."
The extent of the impact was felt across the Google Cloud Platform including G Suite services, Google Compute Engine, App Engine, Cloud Endpoints, Cloud Interconnect, Cloud VPN, Cloud Console, Stackdriver Metrics, Cloud Pub/Sub, Bigquery, regional Cloud Spanner instances, and Cloud Storage regional buckets.
Google is taking steps to mitigate the possibility of such an occurrence happening again.
"Google's emergency response tooling and procedures will be reviewed, updated and tested to ensure that they are robust to network failures of this kind, including our tooling for communicating with the customer base. Furthermore, we will extend our continuous disaster-recovery testing regime to include this and other similarly catastrophic failures," Google said.
A day after the mishap, Alphabet’s stock price took a hit, dropping $67 to $1,036 from $1,103. The stock has yet to recover from the incident. While there’s other factors at play affecting the Company’s stock price, one can’t deny the ramifications that a widespread outage can have on a cloud company, particularly when it’s the world’s most ubiquitous.
