Google Workspace Status Dashboard

INCIDENT REPORT

Summary:

On Saturday, 03 September 2022 21:00 US/Pacific, Google Calendar experienced an issue with a mismatch in Daylight Savings time in Chile for a duration of 3 days, 22 hours, 36 minutes. The issue was mitigated on Wednesday, 07 September 2022 07:36 US/Pacific. We have conducted an internal investigation and are taking steps to improve our service.

Root Cause:

Internet Assigned Numbers Authority (also known as IANA) is a standards organization that oversees global IP address allocation, autonomous system number allocation, root zone management in the Domain Name System (DNS), media types, and other Internet Protocol-related symbols and Internet numbers. As part of its mission, IANA maintains global time zone information [1], commonly known as the time zone database, the Olson time zone database, or just the “tzdb”. This database is not released on a schedule, just when global time zone changes require it.

Chile had a scheduled time zone change on Sunday, 04 September 2022. However, because of elections occurring around the same time, the government decided to postpone the start date of Daylight Savings time from Sunday, 04 September 2022 to Sunday, 11 September 2022. The Chilean government announced this decision on 08 August 2022, less than a month in advance.

IANA released version 2022b of the time zone database on Thursday, 11 August 2022, followed by 2022c (to address technical issues) on Tuesday, 16 August 2022. Google integrates new versions of the time zone database as they become available in a multi-step process involving several teams. It takes approximately one month for time zone database changes to roll out for Google Calendar.

[1] - https://www.iana.org/time-zones

Remediation

Google engineers expedited a roll out of the new time zone definitions on the morning of Wednesday, 07 September 2022 (local Chilean time). This had the following effects:

• The current time in Calendar (the red bar) would be correct.
• New events would appear at the correct time and be in the correct time zone.
• Appointment schedules would appear at the correct time.
• Modified recurring events would appear at the correct time, but instance-specific data (ie. attendee, attendee status, description, location, time, etc.) which did not match the timezone definition would be deleted and new instances would be created in the correct time zone.

Prevention

Google is committed to preventing recurrence by implementing the following action items:

• Implement improvements to the time zone updates rollout process that will reduce the lead time for completion.
• Investigate Calendar interactions with other Workspace products with relation to mismatched time zones to minimize impact in case of a recurrence.

Detailed Description of Impact:

From 03 September 2022 21:00 to 07 September 2022 07:36 US/Pacific, Google Calendar events for all customers in Chile were impacted due to the time change event.

The customer impact included:

• The current time in Calendar (the red bar) was out of sync with actual time and the operating system.
• New and existing events were at the wrong time.
• Appointments were set at the wrong time.
• Appointments created across time zones displayed at the wrong time.
• Updating appointment schedules displayed at the wrong time.
• Mobile applications and devices displayed events at the wrong time.

MINI INCIDENT REPORT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 03 September 2022 21:00

Incident End: 07 September 2022 07:39

Duration: 3 days, 10 hours, 39 minutes

Affected Services and Features:

Google Calendar

Regions/Zones: Chile

Description:

Google Calendar experienced an issue with a mismatch in daylight savings time in Chile. The issue was caused due to the decision of the Government of Chile to delay the time change by a week from 3 September 2022 to 10 September 2022.

Following the announcement of the Chilean Government’s decision on 8 August 2022, Google product teams evaluated options to minimize customer impact. After careful analysis, the Calendar product team decided to not roll out an ad-hoc fix due to the number of variables involved with the fix roll out. So, the Google team published an external blog post on 1 September 2022, outlining the Government’s decision to delay the time change and workarounds to prepare the customers in Chile for the best time change experience.

Upon receiving multiple customer escalations on 5 September 2022, the product team reconvened and decided to roll out the emergency fix to mitigate the impact to our customers.

Customer Impact:

Events in the calendar for the week of 3 September 2022 to 10 September 2022 were not in the right time for users who have events saved in the Chilean time zone or if the user's calendar is set to Chile time zone. Current time displayed on the customers calendars is an hour later than the actual time.