A recent outage of Google services such as Gmail, YouTube, Google Drive, and Maps severely affected the operations of users and organizations across the globe. All customer-facing Google services that require Google OAuth access were unavailable for 47 minutes. The search engine giant stated that the disruption was caused due to a security flaw in its global authentication system.
“The majority of authenticated services experienced similar control plane impact: elevated error rates across all Google Cloud Platform and Google Workspace APIs and Consoles. Products continued to deliver service normally during the incident except where specifically called out below. Most services recovered automatically within a short period of time,” Google said.
The Root Cause
In an official statement, Google stated that its User ID Service maintains a unique identifier for every account and handles authentication credentials for OAuth tokens and cookies. This service rejects users’ requests when it detects outdated data.
“As part of an ongoing migration of the User ID Service to a new quota system, a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0. An existing grace period on enforcing quota restrictions delayed the impact, which eventually expired, triggering automated quota systems to decrease the quota allowed for the User ID service and triggering this incident,” Google explained.
Nearly 15% of users’ requests to Google Cloud Storage (GCS) were affected in the incident, especially the users of OAuth, HMAC, or email authentication.
“The majority of impact was resolved, however, there was lingering impact, for <1% of clients that attempted to finalize resumable uploads that started during the window. These uploads were left in a non-resumable state; the error code GCS returned was retryable, but subsequent retries were unable to make progress, leaving these objects unfinalized,” Google added.
