Jul <var data-var='date'> 7</var>, <var data-var='time'>11:12</var> UTC Resolved - All pending notifications have now been flushed. Notifications scheduled since approximately 02:00 UTC on July 1st were failing to be sent. This affected customers in the US data center making use of the following API features: - Event Triggers; approximately 9000 notifications for calendar events starting/ending will have been delivered late - Scheduler Workflows; a small number of Scheduling Request lifecycle notifications will have been delivered late - Meeting Agents; a small number of scheduled Meeting Agents will not have arrived to their scheduled meetings A postmortem of the incident will take place and be attached to this incident in the next 48 hours. Jul <var data-var='date'> 7</var>, <var data-var='time'>10:20</var> UTC Monitoring - A failing component in our US data center led to scheduled notifications for Event Triggers backing up and not sending. We've resolved the immediate cause, which will lead to notifications being delivered behind schedule. We are monitoring this and investigating any wider impact from the issue. Jul <var data-var='date'> 7</var>, <var data-var='time'>10:12</var> UTC Investigating - We are investigating an issue where Event Trigger notifications are failing to send in our US data center

Event Triggers not sending in US data center

Jun <var data-var='date'>16</var>, <var data-var='time'>09:56</var> UTC Resolved - The mitigation has been completed. Increased API errors were observed from 09:30:30 to 09:31:30 UTC. Jun <var data-var='date'>16</var>, <var data-var='time'>09:25</var> UTC Identified - We've been monitoring an issue on a cache cluster node since shortly after the maintenance at the weekend. As part of mitigation, customers may see increased errors from the API for a short time period. We will begin the mitigation momentarily.

Increased API errors in our US data center

Jun <var data-var='date'>14</var>, <var data-var='time'>09:54</var> UTC Completed - The scheduled maintenance has been completed successfully. We observed disruption at the following times (UTC): Singapore: No disruption observed Australia: 07:30:15 - 07:30:30 UK: No disruption observed Canada: 08:12:15 - 08:13:30 UTC, 08:24:15 - 08:25:30, 09:30:30 - 09:30:45 and 09:33:30 - 09:33:45 Germany: 08:47:30 - 08:49:00 US: 09:16:30 - 09:16:45 If you have any questions, please email support@cronofy.com. Jun <var data-var='date'>14</var>, <var data-var='time'>09:41</var> UTC Verifying - The update is complete in our US data center. We observed disruption from 09:16:30 to 09:16:45 UTC. We also observed some additional disruption in our Canadian data center. This was from 09:30:30 to 09:30:45, and from 09:33:30 to 09:33:45. This extra disruption happened when old database reader nodes were removed from the cluster. This was the only data center where changes to the database cluster were made. Jun <var data-var='date'>14</var>, <var data-var='time'>09:00</var> UTC Update - The update is complete in our Germany data center. We observed disruption from 08:47:30 to 08:49:00 UTC. We are beginning the update in our US data center. Jun <var data-var='date'>14</var>, <var data-var='time'>08:37</var> UTC Update - The update is complete in our Canada data center. We observed disruption from 08:12:15 to 08:13:30 UTC, and then again from 08:24:15 to 08:25:30. We are beginning the update in our Germany data center. Jun <var data-var='date'>14</var>, <var data-var='time'>08:07</var> UTC Update - The update is complete in our UK data center. There was no observed disruption. We are beginning the update in our Canada data center. Jun <var data-var='date'>14</var>, <var data-var='time'>07:43</var> UTC Update - The update is complete in our Australia data center. We observed disruption from 07:30:15 to 07:30:30 UTC. We are beginning the update in our UK data center. Jun <var data-var='date'>14</var>, <var data-var='time'>07:19</var> UTC Update - The update is complete in our Singapore data center. There was no observed disruption. We are beginning the update in our Australia data center. Jun <var data-var='date'>14</var>, <var data-var='time'>07:00</var> UTC In progress - The scheduled maintenance is in progress. We are beginning the update in our Singapore data center. Jun <var data-var='date'> 8</var>, <var data-var='time'>16:00</var> UTC Scheduled - We will be upgrading the instance types used within our cache and database clusters in all data centers. These upgrades require a failover to the new instance types, therefore there will be up to five minutes of disruption when this happens. We will upgrade the instances one data center at a time and will begin the first upgrade shortly after 07:00 UTC. We expect the upgrades to be completed by 10:00 UTC on Sunday, 14th June 2026. We will provide notifications as this maintenance progresses via our status page. If you have any questions, please email support@cronofy.com.

Cache and database cluster instance type upgrades in all data centers

Jun <var data-var='date'> 4</var>, <var data-var='time'>13:05</var> UTC Resolved - Google Meet Meeting Agents are attempting meeting joins successfully again after our further fix earlier today. We shall continue to monitor but consider this incident to be closed. Jun <var data-var='date'> 4</var>, <var data-var='time'>11:00</var> UTC Update - We are now rolling out a further fix which has tested well internally, and will continue to monitor its performance. Jun <var data-var='date'> 4</var>, <var data-var='time'>09:39</var> UTC Update - After monitoring we've seen that the fix has improved performance but we're still seeing elevated cases of failures to join Google Meet Meetings, so are still working improving things further. Jun <var data-var='date'> 3</var>, <var data-var='time'>18:34</var> UTC Update - Google Meet Meeting Agents are making join attempts successfully since the fix release earlier. We will continue to monitor their performance. Jun <var data-var='date'> 3</var>, <var data-var='time'>18:30</var> UTC Update - We are continuing to monitor for any further issues. Jun <var data-var='date'> 3</var>, <var data-var='time'>16:31</var> UTC Monitoring - We are rolling out a fix which has tested positively internally, and will be monitoring success rates Jun <var data-var='date'> 3</var>, <var data-var='time'>15:12</var> UTC Investigating - We have seen that Google Meet Meeting Agents are being rejected from the meetings before being able to join and are investigating the issue. These meetings will fail to be recorded and transcribed.

Google Meet Meeting Agents failing to join

Jun <var data-var='date'> 2</var>, <var data-var='time'>20:29</var> UTC Resolved - Events continue to sync events into Google Calendars as normal. We are still seeing an elevated number of errors between Cronofy and Google Calendar, but are confident that our system are keeping events in sync correctly. The vast majority of events are still syncing almost immediately, while a minority of updates sync after a few seconds. This should have no impact on customers applications, so we are resolving the incident. We will continue to monitor the situation and will raise a new incident if performance declines to a noticeable level. Jun <var data-var='date'> 2</var>, <var data-var='time'>16:55</var> UTC Update - We are continuing to see elevated errors when communicating with Google Calendar. Cronofy is operating normally - event changes are still syncing correctly. The only impact is that some changes may take slightly longer to be reflected in Google Calendars than usual. We are continuing to monitor the situation, but will only update this incident if there is a significant change in either direction. Jun <var data-var='date'> 2</var>, <var data-var='time'>13:40</var> UTC Monitoring - Google's API is returning an increased number of 503 errors when we attempt to sync calendars. This affects a small percentage of calendars, and some users may see delayed calendar syncs as a result. We're continuing to monitor.

Increased error rates for some Google calendar sync operations

May <var data-var='date'>14</var>, <var data-var='time'>10:37</var> UTC Resolved - Between 03:08 UTC and 03:10 UTC we observed increased API error rates in our US data center. During this time, customers may have received 500 Internal Server Error responses. Since that time, error rates have returned to normal levels. These errors were caused by a read-only database server going through an automated recovery process.

Mar <var data-var='date'>13</var>, <var data-var='time'>15:18</var> UTC Resolved - Google has closed their incident and we've seen error rates remain low so are likewise marking this as resolved now. Mar <var data-var='date'>13</var>, <var data-var='time'>14:55</var> UTC Monitoring - Google have identified an issue within their infrastructure which they're tracking at https://www.google.com/appsstatus/dashboard/incidents/KamjizntSiNtRETcBL2Q From our side, we're now seeing very low enough error rates that we don't think that users will experience any noticeable negative impact. We'll keep watching Google's incident until it's fully resolved. Mar <var data-var='date'>13</var>, <var data-var='time'>13:54</var> UTC Update - We are seeing signs of improvement from Google's side -- error rates have improved a great deal but are not yet quite back to normal levels. We continue to monitor the situation. Mar <var data-var='date'>13</var>, <var data-var='time'>13:29</var> UTC Investigating - We've seen that Google's API is returning an increased number of 503 errors when we attempt to sync calendars. This is affecting a small percentage of calendars, and some users may see delayed calendar syncs as a result. We're continuing to monitor and investigate.

Feb <var data-var='date'>27</var>, <var data-var='time'>17:35</var> UTC Resolved - At approximately 12:30 UTC a change was released that prevented events being created in Microsoft calendars if integrated conferencing was being used. A fix was released and the last error observed at approximately 16:05 UTC. Over 90% of the impacted events were successfully synced due to standard retry mechanisms with the platform by 17:00 UTC, we also ran a sweep to proactively retry all remaining events which completed by 17:30 UTC. This incident is now resolved and operations are back to normal. We will be conducting a postmortem and will publish the findings of that next week. Feb <var data-var='date'>27</var>, <var data-var='time'>17:00</var> UTC Update - At approximately 12:30 UTC a change was released that prevented events being created in Microsoft calendars if integrated conferencing was being used. A fix was released and the last error observed at approximately 16:05 UTC. Since that time over 90% of the impacted events have been successfully synced due to standard retry mechanisms with the platform. We are working to enqueue additional attempts for all affected calendars to speed up this process and to be certain nothing has been missed. Feb <var data-var='date'>27</var>, <var data-var='time'>16:28</var> UTC Update - We have identified that affected events were those being written to Microsoft calendars and using the `integrated` conferencing option. The system is already retrying these failures and successfully writing them. We are looking to accelerate this process. Feb <var data-var='date'>27</var>, <var data-var='time'>16:06</var> UTC Identified - Since approximately 12:30 UTC a subset of calendar events failed to be synced to users' calendars. We've identified the source of the issue and released a fix, we are now assessing the impact and actions necessary to resolve the issue for previously impacted calendar events.

Events failing to sync to user calendars

Feb <var data-var='date'>17</var>, <var data-var='time'>14:10</var> UTC Resolved - We have released version 0.22.25 of the Cronofy Chrome extension, fixing the issue. If you have any problems, please first check that both Google Chrome and the Cronofy Chrome extension are up to date. If you have any additional issues then please contact support@cronofy.com. Feb <var data-var='date'>17</var>, <var data-var='time'>13:34</var> UTC Update - We have submitted an updated version of the Chrome Extension to Google which contains a fix for this issue. The updated version will be available as soon as it has been checked and published by Google. We will provide an update once we confirmation of this from Google. Feb <var data-var='date'>17</var>, <var data-var='time'>12:39</var> UTC Identified - A recent Google Chrome update has changed a mechanism that our Gmail extension relies on. We are developing a fix that is compatible with this new mechanism. We will provide an update in a few hours or when we have an updated version of the extension ready. Feb <var data-var='date'>17</var>, <var data-var='time'>12:01</var> UTC Update - We are still working to identify the exact root cause of the issue. We will provide further updates as soon as possible. Feb <var data-var='date'>17</var>, <var data-var='time'>11:22</var> UTC Investigating - We are aware of issues affecting our Google Chrome Extension preventing users from creating scheduling requests in Gmail. We are currently investigating and will provide an update as soon as possible.

Issues scheduling with Google Chrome Extension in Gmail

Feb <var data-var='date'>13</var>, <var data-var='time'>13:44</var> UTC Resolved - Between 12:52 and 12:55 we saw that calls to our API encountered a higher level of 500 Server Error responses than usual We have investigated this and it appears to relate to a period where we saw a large spike in traffic, a rapid drop and another spike in quick succession resulting in some temporary 500 errors. This is now resolved but we will continue monitoring for any further issues.

Elevated API 500 errors in our AU data center

Feb <var data-var='date'>12</var>, <var data-var='time'>10:59</var> UTC Resolved - We have seen that our error rate provisioning MS Teams Conferencing has returned to normal, low levels since 16:10 UTC yesterday. Normal levels of service have resumed, and we're working to improve our monitoring so that similar issues are detected sooner in the future. Feb <var data-var='date'>11</var>, <var data-var='time'>14:59</var> UTC Update - We believe the root cause is a lack of generated URLs for a meeting when we call Microsoft Graph APIs. We are waiting for a response from Microsoft on the issue. Unfortunately the timing of this is out of our hands. We are also working on alternative ways to provision Teams meetings, but this will not provide a short term fix. We are continuing to work on finding a way to resolve, mitigate or provide an more reliable path to Teams provisioning. We will update this incident over the coming days as we make meaningful progress. Feb <var data-var='date'>11</var>, <var data-var='time'>13:31</var> UTC Update - We've found some evidence that other users of the Microsoft Graph API are experiencing the same issue; we're looking for anything that may let us mitigate the impact and we're reaching out to Microsoft for support. Feb <var data-var='date'>11</var>, <var data-var='time'>12:54</var> UTC Update - We are still investigating; we are capturing additional logs to help us narrow down the failing population. Some conferences are being provisioned after automatically retrying, but still not as many as we'd expect. Feb <var data-var='date'>11</var>, <var data-var='time'>12:07</var> UTC Update - We are continuing to investigate - the issue is not impacting all customers but we are not yet seeing a pattern to the domains or events impacted. Feb <var data-var='date'>11</var>, <var data-var='time'>11:29</var> UTC Investigating - We are investigating an elevated level of failures for provisioning MS Teams Conferencing.

MS Teams Conferencing Failures

Feb <var data-var='date'> 1</var>, <var data-var='time'>09:04</var> UTC Completed - The scheduled maintenance has been completed successfully. We observed disruption at the following times (UTC): Singapore: No disruption observed Australia: 06:44:00 - 06:45:15 UK: 07:06:45 - 07:10:15 Canada: 07:32:15 - 07:32:45 Germany: 07:55:15 - 07:55:30 US: 08:26:30 - 08:27:30 If you have any questions, please email support@cronofy.com. Feb <var data-var='date'> 1</var>, <var data-var='time'>08:50</var> UTC Verifying - The update is complete in our US data center. We observed disruption from 08:26:30 to 08:27:30 UTC. Feb <var data-var='date'> 1</var>, <var data-var='time'>08:09</var> UTC Update - The update is complete in our Germany data center. We observed disruption from 07:55:15 to 07:55:30 UTC. We are beginning the update in our US data center. Feb <var data-var='date'> 1</var>, <var data-var='time'>07:45</var> UTC Update - The update is complete in our Canada data center. We observed disruption from 07:32:15 to 07:32:45 UTC. We are beginning the update in our Germany data center. Feb <var data-var='date'> 1</var>, <var data-var='time'>07:27</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'> 1</var>, <var data-var='time'>07:19</var> UTC Update - The update is complete in our UK data center. We observed disruption from 07:06:45 to 07:10:15 UTC. We are beginning the update in our Canada data center. Feb <var data-var='date'> 1</var>, <var data-var='time'>06:56</var> UTC Update - The update is complete in our Australia data center. We observed disruption from 06:44:00 to 06:45:15 UTC. We are beginning the update in our UK data center. Feb <var data-var='date'> 1</var>, <var data-var='time'>06:30</var> UTC Update - The update is complete in our Singapore data center. We observed no disruption during this update. We are beginning the update in our Australia data center. Feb <var data-var='date'> 1</var>, <var data-var='time'>06:01</var> UTC Update - The scheduled maintenance is in progress. We are beginning the update in our Singapore data center. Jan <var data-var='date'>22</var>, <var data-var='time'>09:49</var> UTC Update - We will be undergoing scheduled maintenance during this time. Jan <var data-var='date'>22</var>, <var data-var='time'>09:44</var> UTC Scheduled - We will be performing ElastiCache service updates across all data centers. ElastiCache service updates require a failover, therefore there will be up to 5 minutes of disruption when this happens. We will upgrade each data centers at a time and will begin the first upgrade shortly after 06:00 UTC and expect the upgrades to be completed by 10:00 UTC on Sunday, 1st February 2026. The disruption may result in errors being returned by our application when using the scheduler or our API. Any background processing jobs will automatically be retried. If you have any questions, please email support@cronofy.com.

Elasticache service updates in all our data centers

Jan <var data-var='date'>23</var>, <var data-var='time'>01:19</var> UTC Resolved - Between 22:00 and 01:15 UTC we saw elevated error rates when performing sync operations with Microsoft 365. Microsoft continues to work on a full resolution to the incident with ID MO1221364. However, the error rates we are now observing are back to normal levels. Jan <var data-var='date'>23</var>, <var data-var='time'>00:01</var> UTC Update - We are continuing to monitor the situation, but have not seen any improvement so far Jan <var data-var='date'>22</var>, <var data-var='time'>23:28</var> UTC Monitoring - We believe this is related to a Microsoft incident (ID: MO1221364). We are keeping a close eye on how this progresses Jan <var data-var='date'>22</var>, <var data-var='time'>23:18</var> UTC Investigating - We are currently investigating degraded sync performance for Microsoft 365 as we are experiencing a much higher error rate than normal.

Degraded sync performance for Microsoft 365

Nov <var data-var='date'>30</var>, <var data-var='time'>22:35</var> UTC Resolved - Between 20:45:15 and 20:47:30, and 21:03:15 and 21:03:30 UTC, we observed increased API error rates in our US data center. The first window of these was caused by a database failover, the second of these was caused by a reboot of the database cluster reader node. We took this action in an attempt to alleviate a small but persistent number of database errors that we had seen since the earlier maintenance. During the maintenance, we believed these errors were transient and that they would dissipate throughout the day. As this did not happen, we felt it was necessary to take action now, and that action was to failover, reboot, and then ultimately replace the affected node. Since the replacement, we have not observed the aforementioned database errors. If you have any questions, please email support@cronofy.com. Nov <var data-var='date'>30</var>, <var data-var='time'>21:57</var> UTC Update - We are continuing to monitor post-failover Nov <var data-var='date'>30</var>, <var data-var='time'>21:11</var> UTC Update - We're continuing to monitor the situation. An additional spike in server errors was observed between 21:03:15 and 21:03:30 Nov <var data-var='date'>30</var>, <var data-var='time'>20:51</var> UTC Monitoring - We have chosen to initiate a database failover in our US data center. Because of this, customers may have received a 500 Internal Server error between 20:45:15 and 20:47:30. The vast majority of these errors were observed between 20:45:15 and 20:46:00. We are now monitoring the database post-failover.

Nov <var data-var='date'>30</var>, <var data-var='time'>09:00</var> UTC Completed - The scheduled maintenance has been completed successfully. We observed disruption at the following times (UTC): Singapore: No observed disruption Australia: 07:21:00 - 07:27:00 UK: 07:41:00 - 07:45:00 Canada: 07:53:30 - 07:55:00 Germany: 08:03:00 - 08:07:00 US: 08:20:00 - 08:24:00 If you have any questions, please email support@cronofy.com. Nov <var data-var='date'>30</var>, <var data-var='time'>08:45</var> UTC Verifying - The update is complete in our US data center. We observed disruption from 08:20:00 to 08:24:00 UTC. Nov <var data-var='date'>30</var>, <var data-var='time'>08:11</var> UTC Update - The update is complete in our Germany data center. We observed disruption from 08:03:00 to 08:07:00 UTC. We are beginning the update in our US data center. Nov <var data-var='date'>30</var>, <var data-var='time'>07:59</var> UTC Update - The update is complete in our Canada data center. We observed disruption from 07:53:30 to 07:55:00 UTC. We are beginning the update in our Germany data center. Nov <var data-var='date'>30</var>, <var data-var='time'>07:50</var> UTC Update - The update is complete in our UK data center. We observed disruption from 07:41:00 to 07:45:00 UTC. We are beginning the update in our Canada data center. Nov <var data-var='date'>30</var>, <var data-var='time'>07:37</var> UTC Update - The update is complete in our Australia data center. We observed disruption from 07:21:00 to 07:27:00 UTC. We are beginning the update in our UK data center. Nov <var data-var='date'>30</var>, <var data-var='time'>07:21</var> UTC Update - The update is complete in our Singapore data center. We observed no disruption during this update. We are beginning the update in our Australia data center. Nov <var data-var='date'>30</var>, <var data-var='time'>07:01</var> UTC In progress - The scheduled maintenance is in progress. We are beginning the update in our Singapore data center. Nov <var data-var='date'>24</var>, <var data-var='time'>10:07</var> UTC Scheduled - We will be performing minor PostgreSQL database engine upgrades in our Aurora clusters across all data centers. This process requires a database server restart and failover, therefore there will be up to five minutes of disruption. We will upgrade each data center one at a time and will begin the first upgrade shortly after 07:00 UTC. We expect the upgrades to be completed by 09:00 UTC on Sunday, 30th November 2025. We will provide notifications as this maintenance progresses via our status page. If you have any questions, please email support@cronofy.com.

Minor database upgrade in all data centers

Nov <var data-var='date'>18</var>, <var data-var='time'>18:33</var> UTC Resolved - Between 16:55 UTC and 17:03 UTC we observed increased API error rates in our US data center. During this time customers may have received 401 Unauthorized or 500 Internal Server Error responses. Since that time error rates have returned to normal levels. A postmortem of the incident will take place and be attached to this incident in the coming days. Nov <var data-var='date'>18</var>, <var data-var='time'>18:18</var> UTC Update - Error rates remain at normal levels, and we are continuing to monitor. Nov <var data-var='date'>18</var>, <var data-var='time'>17:37</var> UTC Monitoring - Between 16:55 UTC and 17:03 UTC we observed increased API error rates in our US data center. Error rates returned to normal by 17:03 UTC. We are monitoring and investigating the root cause.

Increased API error rates in our US data center

Nov <var data-var='date'>12</var>, <var data-var='time'>18:56</var> UTC Resolved - Between 18:15 UTC and 18:28 UTC, we observed elevated error rates, and Google Calendar users may have experienced delays with calendar reads and writes. Since 18:28 UTC, all Google Calendar operations have been functioning as normal. Nov <var data-var='date'>12</var>, <var data-var='time'>18:40</var> UTC Monitoring - Google's API error rate has dropped again and has been at normal levels for the past 10 minutes. We'll continue to monitor and confirm this holds out. Nov <var data-var='date'>12</var>, <var data-var='time'>18:31</var> UTC Investigating - We are currently investigating the cause of Google calendar sync degradation

Google calendar sync degraded

Oct <var data-var='date'>23</var>, <var data-var='time'>10:59</var> UTC Resolved - Between 10:12 and 10:18 UTC a change to the way back-end resources are allocated led to a scenario whereby the services processing requests began being forcibly restarted. This was caused by the services rapidly exceeding predefined resource limits and being taken out of service. This resulted in a partial outage that affected a small number of requests. Our internal monitoring alerted us to the issue and the changes were reverted. We're continuing to monitor the situation but consider this incident resolved. Full service was restored at 10:18 UTC. Oct <var data-var='date'>23</var>, <var data-var='time'>10:35</var> UTC Investigating - Customers using our Canadian data center may have experienced interruption of service when accessing both our API and application. This was caused by a recent change and will have presented as HTTP 502 - Gateway Timeout statuses. We have reverted the change and are in the process of assessing the exact scope of the issue.

CA data center - Interruption of service

Oct <var data-var='date'>20</var>, <var data-var='time'>21:58</var> UTC Resolved - On October 20th, AWS's us-east-1 region experienced a significant incident impacting multiple underlying services. This led to the operation of Cronofy's US data center, which is hosted in this AWS region, being severely impacted. None of Cronofy's other data centers were affected at any time. The main impact was felt between 07:15 and 09:15 UTC where many critical components struggled to communicate with AWS's services. Performance was severely degraded throughout this period. We saw some recovery from 09:15 UTC, with the backlog of high priority tasks completed by 09:30 UTC, these are the tasks most likely to have a user-noticeable impact. Some issues remained, mainly the ability to provision additional servers which has been widely reported by other AWS customers. However, we had sufficient capacity to clear the backlog of lower priority tasks by 10:20 UTC. Performance of the US data was back to normal from this time, and we continued to monitor. Throughout the day we ran with an altered configuration to obtain as much capacity as we could from AWS to provide as smooth a service as possible. Our ability to provision additional capacity was restored at 19:15 UTC. While the underlying AWS incident is still open, we are considering this incident to be resolved as we have been able to resume normal operations. A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at support@cronofy.com. Oct <var data-var='date'>20</var>, <var data-var='time'>13:40</var> UTC Update - AWS continue to work towards a full resolution. We remain unable to reliably provision additional capacity in our US data center. However, since the previous message, we have managed to increase the available capacity. Performance of the US data center continues to be normal, and we will continue to monitor. Oct <var data-var='date'>20</var>, <var data-var='time'>10:29</var> UTC Update - Since around 07:15 UTC our US data center has been severely impacted by an ongoing incident in AWS us-east-1, where it is hosted: https://health.aws.amazon.com/health/status At 09:15 UTC we started seeing signs of recovery. As of 09:30 UTC all high priority tasks were cleared which are those with the most visible user impact. Lower priority tasks were cleared by 10:20 UTC. AWS continue to work towards full resolution. We remain unable to reliably provision additional capacity in this data center. We have taken steps to ensure we retain the capacity we already have. Performance of the US data is now back to normal, we continue to monitor whilst the underlying AWS incident is active. Oct <var data-var='date'>20</var>, <var data-var='time'>09:52</var> UTC Monitoring - Since around 07:15 UTC our US data center has been severely impacted by an ongoing incident in AWS us-east-1, where it is hosted: https://health.aws.amazon.com/health/status AWS have identified a potential root cause for the issue and are working on multiple parallel paths to accelerate recovery. At 09:20 UTC we started seeing signs of recovery. As of 09:30 UTC all high priority tasks were cleared which are those with the most visible user impact. We are unable to scale additional capacity to clear lower priority tasks at present, which are more background processes such as polling calendars (Apple) and housekeeping style tasks. This backlog is being processed, but not at a pace that means we are clearing it as quickly as we would like. All other data centers are fully operational.

Major outage in US data center

Sep <var data-var='date'>15</var>, <var data-var='time'>11:49</var> UTC Resolved - Starting at 10:40 UTC we observed an elevated error rate when communicating with Apple's calendar service. Errors returned to regular levels by 11:35 UTC. Sync with Apple calendars may have taken longer both to and from Apple as a result of the underlying errors during this period. Sep <var data-var='date'>15</var>, <var data-var='time'>11:04</var> UTC Monitoring - Since 10:40 UTC we have observed an increased error rate for Apple calendar sync operations. We are still seeing more successes than failures but consider there to be enough failures to mark Apple as degraded. Our retry mechanisms appear to be doing their job, but changes to calendars may take longer to sync both to and from Apple as a result of the underlying errors. We are monitoring the situation.

Apple calendar sync degraded

Aug <var data-var='date'>28</var>, <var data-var='time'>16:44</var> UTC Resolved - Between 15:46 and 16:16 UTC, we saw an increase in response times from the Microsoft 365 Graph API. Since then we have not seen a repeat of this and response times have returned to normal. Aug <var data-var='date'>28</var>, <var data-var='time'>16:26</var> UTC Investigating - We are currently investigating degraded sync performance with Microsoft 365

Aug <var data-var='date'>16</var>, <var data-var='time'>16:28</var> UTC Resolved - Normal operations have continued and we are marking this incident as resolved. A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at support@cronofy.com Aug <var data-var='date'>16</var>, <var data-var='time'>15:55</var> UTC Monitoring - We're monitoring the environment to ensure things continue to run as expected. Aug <var data-var='date'>16</var>, <var data-var='time'>15:53</var> UTC Identified - Investigation has shown that between 15:19 and 15:27 UTC a database issue resulted in a partial loss of service. During this period requests may have returned a 5xx status and failed to process. The database issue has now been resolved and we're moving to monitor the environment. Aug <var data-var='date'>16</var>, <var data-var='time'>15:36</var> UTC Investigating - We are seeing a small number of 5XX errors being returned from our US data center. We are investigating and will update shortly.

Increased error rates in our US data centre

Jul <var data-var='date'>18</var>, <var data-var='time'>16:55</var> UTC Resolved - API success rates have remained stable since our previous message, and Google have resolved their own incident: https://www.google.com/appsstatus/dashboard/incidents/oFcAZTr4EVieF5Fr6Ee9 Between the times of 14:57 UTC - 15:35 UTC, we saw elevated rates of errors, and Google Calendar users may have had slight delays to calendar reads and writes. Since 15:35 UTC everything has been functioning as normal. Jul <var data-var='date'>18</var>, <var data-var='time'>15:48</var> UTC Monitoring - Google's API error rate has dropped again and held at a normal level for the past 10 minutes, and both read and write operations to Google Calendar profiles appear to be back to our normal level of performance. We'll continue to monitor and confirm this holds out. Jul <var data-var='date'>18</var>, <var data-var='time'>15:39</var> UTC Identified - Google is returning a higher rate of 5xx errors when we try to perform token refresh operations. Approximately 1/3rd of our attempts to refresh tokens are seeing this error. The increase in errors started approximately 14:57 UTC. It appears our normal mechanisms are allowing retries to be processed successfully, and limiting the impact to the wider service. Updates to and from Google Calendar profiles may see slight delays during this time. Google appear to have acknowledged the incident: https://www.google.com/appsstatus/dashboard/incidents/oFcAZTr4EVieF5Fr6Ee9 Jul <var data-var='date'>18</var>, <var data-var='time'>15:23</var> UTC Investigating - We've observed an elevated rate of errors from Google's Calendar API and are investigating the scale and impact. Updates to and from Google Calendar profiles may see slight delays during this time. We are monitoring the situation.

Elevated Google Calendar errors

Jul <var data-var='date'>10</var>, <var data-var='time'>18:58</var> UTC Resolved - Microsoft Issue ID EX1112414 remains open however a preliminary root cause has been provided: "A recent service update to an authentication component is unintentionally preventing access for a subset of users, resulting in intermittent service unavailability." We've seen no further issues since our previous update and error rates remain consistent with healthy operations so we're going to consider this incident resolved. Jul <var data-var='date'>10</var>, <var data-var='time'>17:07</var> UTC Update - As of 16:30 UTC we're seeing service levels back at their expected levels and a drastic reduction in error rates. Outlook.com profiles are now authenticating consistently. Microsoft Issue ID EX1112414 remains open however and so we'll continue to monitor things for a while longer in the expectation that this issue is now resolved but just to be sure. Jul <var data-var='date'>10</var>, <var data-var='time'>15:49</var> UTC Update - Error rates remain fairly consistent and Microsoft are in the process of rolling out a fix. We've yet to see the effects of this, are continuing to monitor, and will advise of any further changes. Jul <var data-var='date'>10</var>, <var data-var='time'>12:56</var> UTC Update - Error rates remain fairly consistent. We're continuing to monitor and will advise of any further changes. Jul <var data-var='date'>10</var>, <var data-var='time'>11:05</var> UTC Monitoring - Error rates remain fairly consistent. We're continuing to monitor and will advise of any further changes. Jul <var data-var='date'>10</var>, <var data-var='time'>10:14</var> UTC Identified - We've ascertained that the issue isn't down to changes on our side and are tracking live issue with Outlook.com. Microsoft are reporting on this as Issue ID EX1112414. We can see that not all profiles are affected and that, of those that are, retries do eventually succeed. We'll continue to monitor and advise of any further changes. Jul <var data-var='date'>10</var>, <var data-var='time'>09:45</var> UTC Investigating - We've identified that some Outlook.com profiles may be increased error rates when authenticating with Microsoft's APIs, causing sync failures for those affected. This appears to be related to an ongoing issue with Outlook.com We're investigating the root cause and will update this incident as we understand more.

Increased error rate communicating with Outlook.com calendars

Jul <var data-var='date'> 3</var>, <var data-var='time'>16:07</var> UTC Resolved - Error rates have remained at normal levels and we believe the issue is over. After two increases in error rates, at 14:24 UTC and 14:45 UTC, Google sync has remained stable. A minority of sync operations failed and were retried, eventually completing successfully. Jul <var data-var='date'> 3</var>, <var data-var='time'>15:36</var> UTC Monitoring - Error rates are still low and sync is working as expected. We are continuing to monitor the situation. Jul <var data-var='date'> 3</var>, <var data-var='time'>15:24</var> UTC Update - The error rate has remained low, but we are still investigating and monitoring. Jul <var data-var='date'> 3</var>, <var data-var='time'>15:09</var> UTC Update - Error rates have again returned to normal, and were affecting a fairly small number of operations spread across many accounts. We're continuing to investigate to try and find a pattern. Jul <var data-var='date'> 3</var>, <var data-var='time'>14:50</var> UTC Investigating - We are seeing the issue recur and are continuing to investigate. Jul <var data-var='date'> 3</var>, <var data-var='time'>14:47</var> UTC Monitoring - The error rate has returned to normal. This does not align with any changes or operations on our side, and the error returned from Google suggests this may have been a temporary issue on their side. We are continuing to investigate to try and identify a root cause, as well as monitoring the situation for any further changes. Jul <var data-var='date'> 3</var>, <var data-var='time'>14:36</var> UTC Investigating - We are seeing a sudden increase in Google sync errors and are investigating.

Is Cronofy down?

Cronofy status is UP

Lastest published events

Event Triggers not sending in US data center

Increased API errors in our US data center

Cache and database cluster instance type upgrades in all data centers

Cronofy Services, Components and Regions

Recent Cronofy events

Event Triggers not sending in US data center

Increased API errors in our US data center

Cache and database cluster instance type upgrades in all data centers

Google Meet Meeting Agents failing to join

Increased error rates for some Google calendar sync operations

Increased API errors in our US data center

Get your FREE account now,
5 monitors included!

Is Cronofy down?

Cronofy status is UP

Lastest published events

Event Triggers not sending in US data center

Increased API errors in our US data center

Cache and database cluster instance type upgrades in all data centers

Cronofy Services, Components and Regions

Recent Cronofy events

Event Triggers not sending in US data center

Increased API errors in our US data center

Cache and database cluster instance type upgrades in all data centers

Google Meet Meeting Agents failing to join

Increased error rates for some Google calendar sync operations

Increased API errors in our US data center

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!