Aug <var data-var='date'>17</var>, <var data-var='time'>18:30</var> PDT Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>17</var>, <var data-var='time'>18:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>17</var>, <var data-var='time'>16:17</var> PDT Scheduled - We will be undergoing a database update starting at 9PM EDT for approximately 30 minutes.

Database Update

Jul <var data-var='date'>30</var>, <var data-var='time'>16:44</var> PDT Resolved - This incident has been resolved. Jul <var data-var='date'>30</var>, <var data-var='time'>11:26</var> PDT Investigating - We are observing failures to provision AKS 1.35, 1.36 through CMX and are investigating.

Failures in provisioning AKS on CMX

Jul <var data-var='date'>17</var>, <var data-var='time'>17:00</var> PDT Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>17</var>, <var data-var='time'>16:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>16</var>, <var data-var='time'>09:46</var> PDT Scheduled - We will be performing a database migration during this time. We anticipate an increase in error rates impacting Vendor Portal and Vendor API within a five minute window.

Database update

Jun <var data-var='date'>22</var>, <var data-var='time'>08:57</var> PDT Resolved - At this time we are no longer observing meaningful impact from the upstream incident with Cloudflare. Jun <var data-var='date'>22</var>, <var data-var='time'>07:59</var> PDT Update - We are continuing to monitor updates provided by our upstream networking provider. The immediate impact that we are observing has been significantly reduced in the last 30 minutes. However, the issue is not fully mitigated or resolved in the upstream provider at this time. Jun <var data-var='date'>22</var>, <var data-var='time'>07:32</var> PDT Monitoring - We're observing a higher than expected error rate for requests handled by an upstream networking provider. This may result in a subset of requests being met with high latency and 5xx errors. We are closely monitoring an ongoing incident with the upstream provider at this time for updates and resolution.

Increased error rates through upstream provider

Jun <var data-var='date'> 1</var>, <var data-var='time'>09:39</var> PDT Resolved - The upstream incident with Quay.io which saw failures to Push and Pull images from their registry is improving. We're seeing significantly better success rates for pull operations through the Proxy Registry. At this time impact to Replicated appears mitigated. Jun <var data-var='date'> 1</var>, <var data-var='time'>09:00</var> PDT Update - We're continuing to monitor upstream notices from Redhat regarding the quay.io incident. Latest messaging is that Pulls are operational. Jun <var data-var='date'> 1</var>, <var data-var='time'>08:19</var> PDT Monitoring - The upstream registry has identified the problem and is implementing remediations

Upstream Quay.io degrading push/pull in Image Proxy

May <var data-var='date'>22</var>, <var data-var='time'>09:40</var> PDT Resolved - We're observing now that the upstream dependency issue is resolved and that we're seeing significantly better performance when provisioning OpenShit on CMX. Thank you for your patience with this issue. May <var data-var='date'>22</var>, <var data-var='time'>09:32</var> PDT Monitoring - There are possible delays when provisioning OpenShift on CMX. This relates to an ongoing incident with an upstream dependency. We are closely watching this and will update as conditions improve. Thank you for your patience.

Possible delays in provisioning CMX OpenShift relating to upstream dependencies

May <var data-var='date'>19</var>, <var data-var='time'>15:44</var> PDT Resolved - We've seen consistent improvement to the loading and rendering of https://docs.replicated.com. Thank you for your patience through this issue. May <var data-var='date'>19</var>, <var data-var='time'>14:56</var> PDT Monitoring - https://docs.replicated.com is currently impacted by an incident with an upstream provider. We are closely monitoring this issue and will provide updates here. Thank you for your patience in this matter.

Issues rendering and loading docs.replicated.com

May <var data-var='date'>10</var>, <var data-var='time'>01:19</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>10</var>, <var data-var='time'>01:18</var> PDT Update - We are continuing to investigate this issue. May <var data-var='date'>10</var>, <var data-var='time'>01:17</var> PDT Update - The upstream outage that was affecting this service has been resolved. May <var data-var='date'>10</var>, <var data-var='time'>00:27</var> PDT Update - We are continuing to investigate this issue. May <var data-var='date'>10</var>, <var data-var='time'>00:25</var> PDT Update - We are aware of an outage affecting https://kurl.sh, which prevents access to the documentation and the website that allows building installers yaml. Access to existing installer manifests is unaffected. May <var data-var='date'>10</var>, <var data-var='time'>00:10</var> PDT Investigating - The kURL.sh documentation site is currently experiencing an outage. We are investigating the cause. Installer downloads using kURL are still working and available at this time.

kurl.sh down

May <var data-var='date'> 8</var>, <var data-var='time'>17:34</var> PDT Resolved - This incident has been resolved. May <var data-var='date'> 8</var>, <var data-var='time'>15:57</var> PDT Monitoring - The issue looks to be resolved by azure and we're monitoring the issue. May <var data-var='date'> 8</var>, <var data-var='time'>15:21</var> PDT Investigating - We are currently investigating this issue.

Issues with provisioning AKS in CMX

May <var data-var='date'> 5</var>, <var data-var='time'>19:38</var> PDT Resolved - This incident has been resolved. A fix was released to production. May <var data-var='date'> 5</var>, <var data-var='time'>13:47</var> PDT Identified - We have identified the issue impacting kURL on CMX and are working on a fix. May <var data-var='date'> 5</var>, <var data-var='time'>13:41</var> PDT Update - We are continuing to investigate this issue with provisioning kURL on CMX. We appreciate your patience. May <var data-var='date'> 5</var>, <var data-var='time'>11:04</var> PDT Update - We are continuing to investigate this issue with provisioning kURL on CMX. We appreciate your patience. May <var data-var='date'> 5</var>, <var data-var='time'>10:02</var> PDT Update - We are continuing to investigate this issue and will provide periodic updates on the incident. May <var data-var='date'> 5</var>, <var data-var='time'>09:15</var> PDT Investigating - We are currently investigating this issue

Issues with provisioning kURL in CMX

Apr <var data-var='date'>25</var>, <var data-var='time'>06:29</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>24</var>, <var data-var='time'>10:33</var> PDT Monitoring - CMX is impacted by an upstream incident for Azure AKS. We apologize for the inconvenience and are monitoring for improvements.

CMX Azure AKS is degraded

Feb <var data-var='date'>25</var>, <var data-var='time'>14:01</var> PST Resolved - We are seeing no signs of regression in networking performance or operations of CMX VMS and clusters. This incident is resolved. Feb <var data-var='date'>25</var>, <var data-var='time'>13:35</var> PST Update - CMX operations appear to be working as expected. We are continuing to monitor this situation and the upstream provider. Feb <var data-var='date'>25</var>, <var data-var='time'>13:27</var> PST Monitoring - Internal end to end testing is showing successful operations with VMs and Clusters on CMX Feb <var data-var='date'>25</var>, <var data-var='time'>13:19</var> PST Identified - We believe this relates to a networking issue with an upstream provider which may be transient. We are exercising end to end tests at the moment to observe improvements internally. Feb <var data-var='date'>25</var>, <var data-var='time'>13:12</var> PST Investigating - We are currently investigating this issue.

Issues with Compatibility Matrix

Nov <var data-var='date'>18</var>, <var data-var='time'>07:38</var> PST Resolved - This incident has been resolved. Nov <var data-var='date'>18</var>, <var data-var='time'>07:37</var> PST Update - Internal testing and evaluation of the Replicated platform components shows are are operational. Cloudflare is additionally noting that they believe the incident is resolved, though they are continuing to monitor. Should issues arise again we will keep our customers informed of conditions. Nov <var data-var='date'>18</var>, <var data-var='time'>06:56</var> PST Update - We are continuing to monitor for any further issues. Nov <var data-var='date'>18</var>, <var data-var='time'>06:52</var> PST Monitoring - We are seeing some recovery from the ongoing Cloudflare incident. We are continuing to monitor. Nov <var data-var='date'>18</var>, <var data-var='time'>04:07</var> PST Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>18</var>, <var data-var='time'>04:02</var> PST Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>18</var>, <var data-var='time'>03:59</var> PST Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>18</var>, <var data-var='time'>03:57</var> PST Identified - Replicated platform is currently impacted by ongoing Cloudflare Major Incident (https://www.cloudflarestatus.com/incidents/8gmgl950y3h7)

Replicated platform currently impacted by ongoing Cloudflare Major Incident

Nov <var data-var='date'>12</var>, <var data-var='time'>08:09</var> PST Resolved - We have observed a significant improvement in these resources following actions from an upstream providers networking. This incident is resolved at this time. Nov <var data-var='date'>12</var>, <var data-var='time'>08:00</var> PST Investigating - We are currently investigating this issue.

Possible delays in provisioning CMX resources

Oct <var data-var='date'>30</var>, <var data-var='time'>15:37</var> PDT Resolved - We have tested changes on the roll forward and responses from registry.replicated.com for Helm pulls continue to appear healthy. This incident is resolved. Oct <var data-var='date'>30</var>, <var data-var='time'>14:03</var> PDT Monitoring - We are observing healthy responses from registry.replicated.com at this time. We have identified that error responses observed earlier related to unauthenticated pulls with specific versions of Helm. We are iterating on improvements to mitigate the cause of these problems before rolling back forward. Oct <var data-var='date'>30</var>, <var data-var='time'>12:32</var> PDT Identified - We have issued a rollback for a recent change for registry.replicated.com and are monitoring for improvement Oct <var data-var='date'>30</var>, <var data-var='time'>12:20</var> PDT Investigating - We are currently investigating this issue

Increased HTTP 5xx rate for registry.replicated.com

Oct <var data-var='date'>20</var>, <var data-var='time'>17:42</var> PDT Resolved - AWS is stating at this time that their incident affecting us-east-1 is now resolved. Through careful monitoring and testing we observe that EKS on CMX is working once again Oct <var data-var='date'>20</var>, <var data-var='time'>15:23</var> PDT Update - We are beginning to see recovery for EKS on Compatibility Matrix. We will continue to monitor this and update as conditions change. Oct <var data-var='date'>20</var>, <var data-var='time'>07:07</var> PDT Monitoring - We have identified that provisioning new EKS clusters on CMX has a high failure rate. This appears correlate with an ongoing incident impacting multiple services at AWS. We are monitoring this situation closely.

Compatibility Matrix clusters on EKS are impacted by ongoing incident with AWS

Oct <var data-var='date'>20</var>, <var data-var='time'>11:13</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>09:50</var> PDT Monitoring - A fix has been rolled out, and we are monitoring the results. Vendors may need to refresh the webpage. Oct <var data-var='date'>20</var>, <var data-var='time'>09:22</var> PDT Identified - The issue has been identified and a fix is being deployed. Oct <var data-var='date'>20</var>, <var data-var='time'>08:22</var> PDT Investigating - We are currently investigating this issue.

Reports that updates within Vendor Portal are unsuccessful

Oct <var data-var='date'>16</var>, <var data-var='time'>14:55</var> PDT Resolved - Monitoring appears to show impact has been mitigated at this time. Oct <var data-var='date'>16</var>, <var data-var='time'>14:36</var> PDT Monitoring - We've issued a rollback for the affected platform component and are monitoring for full recovery. Oct <var data-var='date'>16</var>, <var data-var='time'>14:31</var> PDT Identified - We believe we have identified the source of the issue and are working to remediate Oct <var data-var='date'>16</var>, <var data-var='time'>14:28</var> PDT Investigating - We are currently investigating causes of this issue

Increased rate of HTTP 500's reported from replicated.app

Sep <var data-var='date'>26</var>, <var data-var='time'>15:10</var> PDT Resolved - This incident has been resolved. Sep <var data-var='date'>26</var>, <var data-var='time'>14:21</var> PDT Monitoring - We are seeing network performance improvements now, but are still monitoring this issue. Sep <var data-var='date'>26</var>, <var data-var='time'>14:19</var> PDT Update - We are seeing network performance improvements now, but are still monitoring this issue. Sep <var data-var='date'>26</var>, <var data-var='time'>10:05</var> PDT Investigating - We're experiencing degraded network performance for Compatibility Matrix, and are currently looking into it

Slow performance for VM based cluster installations

Sep <var data-var='date'>25</var>, <var data-var='time'>15:06</var> PDT Resolved - We haven't seen any further occurrences of this issue and are confident that the situation is now resolved. Sep <var data-var='date'>25</var>, <var data-var='time'>13:24</var> PDT Update - We are seeing network performance improvements now, but are still monitoring this issue. Sep <var data-var='date'>25</var>, <var data-var='time'>09:47</var> PDT Investigating - We're experiencing degraded network performance for Compatibility Matrix, and are currently looking into it

Sep <var data-var='date'>23</var>, <var data-var='time'>08:29</var> PDT Resolved - We're continuing to observe consistent network performance and download success from EU regions communicating with Replicated platform components at this time. Sep <var data-var='date'>22</var>, <var data-var='time'>16:42</var> PDT Update - We are beginning to observe improved performance for downloads with workloads in the EU at this time. kURL and Embedded Cluster installations on CMX appear to be completing successfully. Sep <var data-var='date'>22</var>, <var data-var='time'>14:16</var> PDT Update - We are continuing to monitor for any further issues. Sep <var data-var='date'>22</var>, <var data-var='time'>14:13</var> PDT Monitoring - We're experiencing significant network performance issues from customers and workloads in EU regions when communicating with Replicated platform. Sep <var data-var='date'>22</var>, <var data-var='time'>13:57</var> PDT Investigating - We are currently investigating this issue.

Performance degradation experienced for EU based workloads involving Replicated platform components.

Sep <var data-var='date'>19</var>, <var data-var='time'>14:40</var> PDT Resolved - Google Cloud states this service disruption has been resolved. We have validated this recovery in CMX and observe GKE clusters creating successfully at this time. Sep <var data-var='date'>19</var>, <var data-var='time'>12:53</var> PDT Monitoring - Creating GKE clusters through Compatibility Matrix appears to be impacted by an ongoing service disruption with GCP for GCE VMs in us-central. We are closely monitoring this incident with Google

Increased GKE creation failure rates in Compatibility Matrix

Sep <var data-var='date'>18</var>, <var data-var='time'>11:42</var> PDT Resolved - Observed issues with an upstream provider have been resolved. The API and the Registry are behaving normally at this time. Sep <var data-var='date'>18</var>, <var data-var='time'>10:02</var> PDT Monitoring - After further review of the increase in HTTP 5xx errors, we've identified that these failed requests are generally coming through an upstream provider in a particular point of presence. We are monitoring ongoing issues with our upstream provider and will update here as conditions change. Sep <var data-var='date'>18</var>, <var data-var='time'>08:48</var> PDT Investigating - We currently investigating this issue

Increased HTTP 5xx rate for api.replicated.com

Sep <var data-var='date'> 4</var>, <var data-var='time'>17:52</var> PDT Completed - The scheduled maintenance has been completed. Sep <var data-var='date'> 4</var>, <var data-var='time'>17:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'> 2</var>, <var data-var='time'>15:09</var> PDT Scheduled - We will be undergoing maintenance during this time. We expect a brief increase in Vendor API latency during this time, especially when performing read operations involving instance metrics, network events, or registry logs. Additionally, instance metric and other event reporting will be delayed.

Metrics Database Maintenance

Aug <var data-var='date'>21</var>, <var data-var='time'>13:57</var> PDT Resolved - The upstream incident has been resolved and we no longer see degradation. Aug <var data-var='date'>21</var>, <var data-var='time'>12:36</var> PDT Monitoring - Services appear to be recovering. We will continue to monitor the issue. Aug <var data-var='date'>21</var>, <var data-var='time'>11:38</var> PDT Update - We suspect that this is related to an ongoing Cloudflare incident: https://www.cloudflarestatus.com/incidents/d9n3g1vnxdd2 Aug <var data-var='date'>21</var>, <var data-var='time'>11:37</var> PDT Update - We are continuing to investigate this issue. Aug <var data-var='date'>21</var>, <var data-var='time'>11:36</var> PDT Investigating - We are investigating an issue where many services appear to have degraded performance or failed responses.

Is Replicated down?

Replicated status is UP

Lastest published events

Database Update

Failures in provisioning AKS on CMX

Database update

Replicated Services, Components and Regions

Recent Replicated events

Database Update

Failures in provisioning AKS on CMX

Database update

Increased error rates through upstream provider

Upstream Quay.io degrading push/pull in Image Proxy

Possible delays in provisioning CMX OpenShift relating to upstream dependencies