Jan <var data-var='date'>18</var>, <var data-var='time'>11:00</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>18</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>12</var>, <var data-var='time'>17:08</var> UTC Scheduled - We will be undergoing scheduled system maintenance during this time. Access to the Cloud 66 dashboard may be intermittently affected. Customer applications are not affected.

Core System Maintenance

Jul <var data-var='date'> 4</var>, <var data-var='time'>21:00</var> UTC Completed - The scheduled maintenance has been completed. Jul <var data-var='date'> 4</var>, <var data-var='time'>18:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'> 4</var>, <var data-var='time'>12:08</var> UTC Scheduled - We will be performing emergency backend maintenance during this period. As a result, some operations may experience delays or temporary unavailability.

Emergency Backend Maintenance

Jun <var data-var='date'>12</var>, <var data-var='time'>20:30</var> UTC Resolved - The underlying issue with GCP seems to be resolved, and we are not seeing further issues with the logins and other systems. Jun <var data-var='date'>12</var>, <var data-var='time'>19:09</var> UTC Investigating - Google authentication issues is having issues causing login and access issues for our systems and customers. We are investigating the impact and mitigation strategies.

Google login issues

Jan <var data-var='date'> 2</var>, <var data-var='time'>15:45</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'> 2</var>, <var data-var='time'>12:25</var> UTC Investigating - We are currently investigating this issue. Jan <var data-var='date'> 2</var>, <var data-var='time'>12:25</var> UTC Identified - The issue has been identified and a fix is being implemented. Jan <var data-var='date'> 2</var>, <var data-var='time'>12:24</var> UTC Investigating - We are aware of problems when scaling/creating Kubernetes clusters. This appears to be caused by new rate limits imposed by docker hub. The team are working on a solution.

Kubernetes Cluster Scale/Creation

Jun <var data-var='date'>29</var>, <var data-var='time'>16:27</var> UTC Completed - The scheduled maintenance of the metrics subsystem is now done. Jun <var data-var='date'>29</var>, <var data-var='time'>15:58</var> UTC In progress - Scheduled maintenance of our metrics subsystem is now in progress. Jun <var data-var='date'>26</var>, <var data-var='time'>16:45</var> UTC Scheduled - Will be undergoing an emergency maintenance for our metrics stack. This will have no impact on deployments or your applications. Metrics and Autoscaling will not be available during this time.

Emergency Metrics Maintenance

Jun <var data-var='date'>28</var>, <var data-var='time'>13:46</var> UTC Resolved - This incident has now been resolved. Jun <var data-var='date'>28</var>, <var data-var='time'>01:50</var> UTC Monitoring - We have identified the issue causing the deployments to be stuck and fixed the issue. This was caused by a database failure at our cloud provider which we are investigating and monitoring. Jun <var data-var='date'>28</var>, <var data-var='time'>01:35</var> UTC Investigating - We are currently investigating a number of stuck deployments.

Stuck Deployments

Jun <var data-var='date'>10</var>, <var data-var='time'>08:00</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>10</var>, <var data-var='time'>08:00</var> UTC Update - We are continuing to verify the maintenance items. Jun <var data-var='date'> 9</var>, <var data-var='time'>08:37</var> UTC Verifying - Verification is currently underway for the maintenance items. Jun <var data-var='date'> 9</var>, <var data-var='time'>07:15</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>23</var>, <var data-var='time'>15:09</var> UTC Scheduled - We will be undergoing scheduled system maintenance during this time. Access to the Cloud 66 dashboard may be intermittently affected. Customer applications are not affected.

Core System Updates

May <var data-var='date'>20</var>, <var data-var='time'>15:07</var> UTC Resolved - The CustomConfig git backend migration has been completed May <var data-var='date'>14</var>, <var data-var='time'>11:48</var> UTC Update - Applications are able to be deployed, but the CustomConfig backend is not available for the moment. May <var data-var='date'>13</var>, <var data-var='time'>08:27</var> UTC Investigating - We are aware of issues with our CustomConfig git repositories. This issue impacts direct git access to the CustomConfig git repository. Direct access to CustomConfig pages via the web dashboard and API is not affected. We are investigating the root cause of the problem.

CustomConfig git access

Mar <var data-var='date'>19</var>, <var data-var='time'>10:13</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>19</var>, <var data-var='time'>10:09</var> UTC Update - We are continuing to investigate this issue. Mar <var data-var='date'>19</var>, <var data-var='time'>10:07</var> UTC Investigating - We are investigating issues affecting some production systems.

Production Issues

Mar <var data-var='date'>16</var>, <var data-var='time'>08:17</var> UTC Completed - The scheduled maintenance has been completed. Mar <var data-var='date'>16</var>, <var data-var='time'>08:03</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>12</var>, <var data-var='time'>10:25</var> UTC Scheduled - We will be undergoing scheduled maintenance during this time.

Core Service Maintenance

Jan <var data-var='date'>20</var>, <var data-var='time'>12:48</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>20</var>, <var data-var='time'>10:34</var> UTC Verifying - Verification is currently underway for the maintenance items. Jan <var data-var='date'>20</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>10</var>, <var data-var='time'>10:34</var> UTC Scheduled - We will be undergoing scheduled maintenance at this time. During this time the Cloud 66 application will be intermittently unavailable for customer deployments. Customer's servers will not be impacted

May <var data-var='date'> 9</var>, <var data-var='time'>15:21</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 9</var>, <var data-var='time'>11:45</var> UTC Monitoring - Github is experiencing some API outages, deployments don't currently appear to be affected. We are monitoring (https://www.githubstatus.com/)

Github API Outage

Apr <var data-var='date'>27</var>, <var data-var='time'>08:22</var> UTC Resolved - Maestro installations are working as expected again Apr <var data-var='date'>26</var>, <var data-var='time'>16:37</var> UTC Monitoring - We are waiting on Google resolutions Apr <var data-var='date'>26</var>, <var data-var='time'>16:03</var> UTC Update - Due to a major outage in Google Cloud (https://status.cloud.google.com/regional/europe) - Kubernetes installations may fail on installations on servers located in the same region, regardless of the cloud, due to Kubernetes images themselves being hosted on Google Cloud Apr <var data-var='date'>26</var>, <var data-var='time'>15:36</var> UTC Investigating - Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

Multiple Google Cloud services in the europe-west9 region are impacted

Oct <var data-var='date'> 7</var>, <var data-var='time'>22:26</var> UTC Resolved - Issue confirmed to be fixed by GitHub. Oct <var data-var='date'> 6</var>, <var data-var='time'>14:52</var> UTC Identified - The issue has been identified and a fix is being implemented and this should resolve the issue for most users. If the error is still persisting, please use the workaround provided: 1. Remove the Github installation. 2. Manually configure SSH key access to Github, please see the link below. https://help.cloud66.com/rails/how-to-guides/common-tools/access-your-code#manually-configuring-github-access Oct <var data-var='date'> 6</var>, <var data-var='time'>13:07</var> UTC Investigating - Currently, GitHub is having an issue with listing repositories while using the Cloud 66 Github app. As a workaround while this issue is ongoing. 1. Remove the Github installation. 2. Manually configure SSH key access to Github, please see the link below. https://help.cloud66.com/rails/how-to-guides/common-tools/access-your-code#manually-configuring-github-access

Github Not Listing Repositories

Sep <var data-var='date'>19</var>, <var data-var='time'>14:50</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'>16</var>, <var data-var='time'>11:47</var> UTC Monitoring - The team has completed the first part of the mitigation strategy for intermittent slow Buildgrid pushes. We are now monitoring ongoing performance. Sep <var data-var='date'>15</var>, <var data-var='time'>17:03</var> UTC Update - Our engineering team is continuing to look into mitigations for this issue. Sep <var data-var='date'>14</var>, <var data-var='time'>20:15</var> UTC Identified - The team is aware that some Buildgrid pushes are taking longer than normal, and are working on a mitigation strategy

Subset of Buildgrid Image pushes slower than normal

Sep <var data-var='date'> 9</var>, <var data-var='time'>08:51</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'> 9</var>, <var data-var='time'>07:20</var> UTC Identified - The issue has been identified and a fix is being implemented.

Production Agent Reporting Outage

Sep <var data-var='date'> 6</var>, <var data-var='time'>11:57</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'> 6</var>, <var data-var='time'>11:53</var> UTC Investigating - GitHub is having an outage at this time. This may adversely affect Deployments.

Github outage

Sep <var data-var='date'> 5</var>, <var data-var='time'>08:21</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>31</var>, <var data-var='time'>09:15</var> UTC Monitoring - We are continuing to monitor this incident Aug <var data-var='date'>30</var>, <var data-var='time'>15:11</var> UTC Update - Azure is recommending that affected customers reboot their servers to obtain an updated DHCP lease to resolve this issue Aug <var data-var='date'>30</var>, <var data-var='time'>09:09</var> UTC Identified - From Azure (https://status.azure.com/en-gb/status) Starting at approximately 06:00 UTC on 30 Aug 2022, a number of customers running Ubuntu 18.04 (bionic) VMs recently upgraded to systemd version 237-3ubuntu10.54 reported experiencing DNS errors when trying to access their resources. Reports of this issue are confined to this single Ubuntu version.

Azure reporting DNS problems (affecting Ubuntu 18.04)

Jul <var data-var='date'>16</var>, <var data-var='time'>10:19</var> UTC Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>16</var>, <var data-var='time'>10:14</var> UTC Update - Scheduled maintenance is still in progress. We will provide updates as necessary. Jul <var data-var='date'>16</var>, <var data-var='time'>09:37</var> UTC Update - Scheduled maintenance is still in progress. We will provide updates as necessary. Jul <var data-var='date'>16</var>, <var data-var='time'>07:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>14</var>, <var data-var='time'>16:26</var> UTC Update - We will be undergoing scheduled maintenance during this time. Jul <var data-var='date'>14</var>, <var data-var='time'>16:25</var> UTC Update - We will be undergoing scheduled maintenance during this time. Jul <var data-var='date'>14</var>, <var data-var='time'>14:46</var> UTC Update - We will be undergoing scheduled maintenance during this time. Jul <var data-var='date'>14</var>, <var data-var='time'>14:45</var> UTC Scheduled - We will be undergoing scheduled maintenance at this time. During this time the Cloud 66 application will be intermittently unavailable for customer deployments.

Jun <var data-var='date'>11</var>, <var data-var='time'>09:26</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>11</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>11</var>, <var data-var='time'>08:49</var> UTC Scheduled - We will be undertaking a scheduled maintenance. During this time, the core services will be unavailable but there will be no impact on your applications, traffics and scheduled backups.

Is Cloud 66 down?

Cloud 66 status is UP

Lastest published events

Core System Maintenance

Emergency Backend Maintenance

Google login issues

Cloud 66 Services, Components and Regions

Recent Cloud 66 events

Core System Maintenance

Emergency Backend Maintenance

Google login issues

Kubernetes Cluster Scale/Creation

Emergency Metrics Maintenance

Stuck Deployments