May <var data-var='date'>22</var>, <var data-var='time'>09:40</var> PDT Resolved - We're observing now that the upstream dependency issue is resolved and that we're seeing significantly better performance when provisioning OpenShit on CMX. Thank you for your patience with this issue. May <var data-var='date'>22</var>, <var data-var='time'>09:32</var> PDT Monitoring - There are possible delays when provisioning OpenShift on CMX. This relates to an ongoing incident with an upstream dependency. We are closely watching this and will update as conditions improve. Thank you for your patience.

Possible delays in provisioning CMX OpenShift relating to upstream dependencies

May <var data-var='date'>19</var>, <var data-var='time'>15:44</var> PDT Resolved - We've seen consistent improvement to the loading and rendering of https://docs.replicated.com. Thank you for your patience through this issue. May <var data-var='date'>19</var>, <var data-var='time'>14:56</var> PDT Monitoring - https://docs.replicated.com is currently impacted by an incident with an upstream provider. We are closely monitoring this issue and will provide updates here. Thank you for your patience in this matter.

Issues rendering and loading docs.replicated.com

May <var data-var='date'>10</var>, <var data-var='time'>01:19</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>10</var>, <var data-var='time'>01:18</var> PDT Update - We are continuing to investigate this issue. May <var data-var='date'>10</var>, <var data-var='time'>01:17</var> PDT Update - The upstream outage that was affecting this service has been resolved. May <var data-var='date'>10</var>, <var data-var='time'>00:27</var> PDT Update - We are continuing to investigate this issue. May <var data-var='date'>10</var>, <var data-var='time'>00:25</var> PDT Update - We are aware of an outage affecting https://kurl.sh, which prevents access to the documentation and the website that allows building installers yaml. Access to existing installer manifests is unaffected. May <var data-var='date'>10</var>, <var data-var='time'>00:10</var> PDT Investigating - The kURL.sh documentation site is currently experiencing an outage. We are investigating the cause. Installer downloads using kURL are still working and available at this time.

kurl.sh down

May <var data-var='date'> 8</var>, <var data-var='time'>17:34</var> PDT Resolved - This incident has been resolved. May <var data-var='date'> 8</var>, <var data-var='time'>15:57</var> PDT Monitoring - The issue looks to be resolved by azure and we're monitoring the issue. May <var data-var='date'> 8</var>, <var data-var='time'>15:21</var> PDT Investigating - We are currently investigating this issue.

Issues with provisioning AKS in CMX

May <var data-var='date'> 5</var>, <var data-var='time'>19:38</var> PDT Resolved - This incident has been resolved. A fix was released to production. May <var data-var='date'> 5</var>, <var data-var='time'>13:47</var> PDT Identified - We have identified the issue impacting kURL on CMX and are working on a fix. May <var data-var='date'> 5</var>, <var data-var='time'>13:41</var> PDT Update - We are continuing to investigate this issue with provisioning kURL on CMX. We appreciate your patience. May <var data-var='date'> 5</var>, <var data-var='time'>11:04</var> PDT Update - We are continuing to investigate this issue with provisioning kURL on CMX. We appreciate your patience. May <var data-var='date'> 5</var>, <var data-var='time'>10:02</var> PDT Update - We are continuing to investigate this issue and will provide periodic updates on the incident. May <var data-var='date'> 5</var>, <var data-var='time'>09:15</var> PDT Investigating - We are currently investigating this issue

Issues with provisioning kURL in CMX

Apr <var data-var='date'>25</var>, <var data-var='time'>06:29</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>24</var>, <var data-var='time'>10:33</var> PDT Monitoring - CMX is impacted by an upstream incident for Azure AKS. We apologize for the inconvenience and are monitoring for improvements.

CMX Azure AKS is degraded

Feb <var data-var='date'>25</var>, <var data-var='time'>14:01</var> PST Resolved - We are seeing no signs of regression in networking performance or operations of CMX VMS and clusters. This incident is resolved. Feb <var data-var='date'>25</var>, <var data-var='time'>13:35</var> PST Update - CMX operations appear to be working as expected. We are continuing to monitor this situation and the upstream provider. Feb <var data-var='date'>25</var>, <var data-var='time'>13:27</var> PST Monitoring - Internal end to end testing is showing successful operations with VMs and Clusters on CMX Feb <var data-var='date'>25</var>, <var data-var='time'>13:19</var> PST Identified - We believe this relates to a networking issue with an upstream provider which may be transient. We are exercising end to end tests at the moment to observe improvements internally. Feb <var data-var='date'>25</var>, <var data-var='time'>13:12</var> PST Investigating - We are currently investigating this issue.

Issues with Compatibility Matrix

Nov <var data-var='date'>18</var>, <var data-var='time'>07:38</var> PST Resolved - This incident has been resolved. Nov <var data-var='date'>18</var>, <var data-var='time'>07:37</var> PST Update - Internal testing and evaluation of the Replicated platform components shows are are operational. Cloudflare is additionally noting that they believe the incident is resolved, though they are continuing to monitor. Should issues arise again we will keep our customers informed of conditions. Nov <var data-var='date'>18</var>, <var data-var='time'>06:56</var> PST Update - We are continuing to monitor for any further issues. Nov <var data-var='date'>18</var>, <var data-var='time'>06:52</var> PST Monitoring - We are seeing some recovery from the ongoing Cloudflare incident. We are continuing to monitor. Nov <var data-var='date'>18</var>, <var data-var='time'>04:07</var> PST Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>18</var>, <var data-var='time'>04:02</var> PST Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>18</var>, <var data-var='time'>03:59</var> PST Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>18</var>, <var data-var='time'>03:57</var> PST Identified - Replicated platform is currently impacted by ongoing Cloudflare Major Incident (https://www.cloudflarestatus.com/incidents/8gmgl950y3h7)

Replicated platform currently impacted by ongoing Cloudflare Major Incident

Nov <var data-var='date'>12</var>, <var data-var='time'>08:09</var> PST Resolved - We have observed a significant improvement in these resources following actions from an upstream providers networking. This incident is resolved at this time. Nov <var data-var='date'>12</var>, <var data-var='time'>08:00</var> PST Investigating - We are currently investigating this issue.

Possible delays in provisioning CMX resources

Oct <var data-var='date'>30</var>, <var data-var='time'>15:37</var> PDT Resolved - We have tested changes on the roll forward and responses from registry.replicated.com for Helm pulls continue to appear healthy. This incident is resolved. Oct <var data-var='date'>30</var>, <var data-var='time'>14:03</var> PDT Monitoring - We are observing healthy responses from registry.replicated.com at this time. We have identified that error responses observed earlier related to unauthenticated pulls with specific versions of Helm. We are iterating on improvements to mitigate the cause of these problems before rolling back forward. Oct <var data-var='date'>30</var>, <var data-var='time'>12:32</var> PDT Identified - We have issued a rollback for a recent change for registry.replicated.com and are monitoring for improvement Oct <var data-var='date'>30</var>, <var data-var='time'>12:20</var> PDT Investigating - We are currently investigating this issue

Increased HTTP 5xx rate for registry.replicated.com

Oct <var data-var='date'>20</var>, <var data-var='time'>17:42</var> PDT Resolved - AWS is stating at this time that their incident affecting us-east-1 is now resolved. Through careful monitoring and testing we observe that EKS on CMX is working once again Oct <var data-var='date'>20</var>, <var data-var='time'>15:23</var> PDT Update - We are beginning to see recovery for EKS on Compatibility Matrix. We will continue to monitor this and update as conditions change. Oct <var data-var='date'>20</var>, <var data-var='time'>07:07</var> PDT Monitoring - We have identified that provisioning new EKS clusters on CMX has a high failure rate. This appears correlate with an ongoing incident impacting multiple services at AWS. We are monitoring this situation closely.

Compatibility Matrix clusters on EKS are impacted by ongoing incident with AWS

Oct <var data-var='date'>20</var>, <var data-var='time'>11:13</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>09:50</var> PDT Monitoring - A fix has been rolled out, and we are monitoring the results. Vendors may need to refresh the webpage. Oct <var data-var='date'>20</var>, <var data-var='time'>09:22</var> PDT Identified - The issue has been identified and a fix is being deployed. Oct <var data-var='date'>20</var>, <var data-var='time'>08:22</var> PDT Investigating - We are currently investigating this issue.

Reports that updates within Vendor Portal are unsuccessful

Oct <var data-var='date'>16</var>, <var data-var='time'>14:55</var> PDT Resolved - Monitoring appears to show impact has been mitigated at this time. Oct <var data-var='date'>16</var>, <var data-var='time'>14:36</var> PDT Monitoring - We've issued a rollback for the affected platform component and are monitoring for full recovery. Oct <var data-var='date'>16</var>, <var data-var='time'>14:31</var> PDT Identified - We believe we have identified the source of the issue and are working to remediate Oct <var data-var='date'>16</var>, <var data-var='time'>14:28</var> PDT Investigating - We are currently investigating causes of this issue

Increased rate of HTTP 500's reported from replicated.app

Sep <var data-var='date'>26</var>, <var data-var='time'>15:10</var> PDT Resolved - This incident has been resolved. Sep <var data-var='date'>26</var>, <var data-var='time'>14:21</var> PDT Monitoring - We are seeing network performance improvements now, but are still monitoring this issue. Sep <var data-var='date'>26</var>, <var data-var='time'>14:19</var> PDT Update - We are seeing network performance improvements now, but are still monitoring this issue. Sep <var data-var='date'>26</var>, <var data-var='time'>10:05</var> PDT Investigating - We're experiencing degraded network performance for Compatibility Matrix, and are currently looking into it

Slow performance for VM based cluster installations

Sep <var data-var='date'>25</var>, <var data-var='time'>15:06</var> PDT Resolved - We haven't seen any further occurrences of this issue and are confident that the situation is now resolved. Sep <var data-var='date'>25</var>, <var data-var='time'>13:24</var> PDT Update - We are seeing network performance improvements now, but are still monitoring this issue. Sep <var data-var='date'>25</var>, <var data-var='time'>09:47</var> PDT Investigating - We're experiencing degraded network performance for Compatibility Matrix, and are currently looking into it

Sep <var data-var='date'>23</var>, <var data-var='time'>08:29</var> PDT Resolved - We're continuing to observe consistent network performance and download success from EU regions communicating with Replicated platform components at this time. Sep <var data-var='date'>22</var>, <var data-var='time'>16:42</var> PDT Update - We are beginning to observe improved performance for downloads with workloads in the EU at this time. kURL and Embedded Cluster installations on CMX appear to be completing successfully. Sep <var data-var='date'>22</var>, <var data-var='time'>14:16</var> PDT Update - We are continuing to monitor for any further issues. Sep <var data-var='date'>22</var>, <var data-var='time'>14:13</var> PDT Monitoring - We're experiencing significant network performance issues from customers and workloads in EU regions when communicating with Replicated platform. Sep <var data-var='date'>22</var>, <var data-var='time'>13:57</var> PDT Investigating - We are currently investigating this issue.

Performance degradation experienced for EU based workloads involving Replicated platform components.

Sep <var data-var='date'>19</var>, <var data-var='time'>14:40</var> PDT Resolved - Google Cloud states this service disruption has been resolved. We have validated this recovery in CMX and observe GKE clusters creating successfully at this time. Sep <var data-var='date'>19</var>, <var data-var='time'>12:53</var> PDT Monitoring - Creating GKE clusters through Compatibility Matrix appears to be impacted by an ongoing service disruption with GCP for GCE VMs in us-central. We are closely monitoring this incident with Google

Increased GKE creation failure rates in Compatibility Matrix

Sep <var data-var='date'>18</var>, <var data-var='time'>11:42</var> PDT Resolved - Observed issues with an upstream provider have been resolved. The API and the Registry are behaving normally at this time. Sep <var data-var='date'>18</var>, <var data-var='time'>10:02</var> PDT Monitoring - After further review of the increase in HTTP 5xx errors, we've identified that these failed requests are generally coming through an upstream provider in a particular point of presence. We are monitoring ongoing issues with our upstream provider and will update here as conditions change. Sep <var data-var='date'>18</var>, <var data-var='time'>08:48</var> PDT Investigating - We currently investigating this issue

Increased HTTP 5xx rate for api.replicated.com

Sep <var data-var='date'> 4</var>, <var data-var='time'>17:52</var> PDT Completed - The scheduled maintenance has been completed. Sep <var data-var='date'> 4</var>, <var data-var='time'>17:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'> 2</var>, <var data-var='time'>15:09</var> PDT Scheduled - We will be undergoing maintenance during this time. We expect a brief increase in Vendor API latency during this time, especially when performing read operations involving instance metrics, network events, or registry logs. Additionally, instance metric and other event reporting will be delayed.

Metrics Database Maintenance

Aug <var data-var='date'>21</var>, <var data-var='time'>13:57</var> PDT Resolved - The upstream incident has been resolved and we no longer see degradation. Aug <var data-var='date'>21</var>, <var data-var='time'>12:36</var> PDT Monitoring - Services appear to be recovering. We will continue to monitor the issue. Aug <var data-var='date'>21</var>, <var data-var='time'>11:38</var> PDT Update - We suspect that this is related to an ongoing Cloudflare incident: https://www.cloudflarestatus.com/incidents/d9n3g1vnxdd2 Aug <var data-var='date'>21</var>, <var data-var='time'>11:37</var> PDT Update - We are continuing to investigate this issue. Aug <var data-var='date'>21</var>, <var data-var='time'>11:36</var> PDT Investigating - We are investigating an issue where many services appear to have degraded performance or failed responses.

Multiple services are degraded

Jul <var data-var='date'>15</var>, <var data-var='time'>16:45</var> PDT Resolved - Service has been returned to normal. Jul <var data-var='date'>15</var>, <var data-var='time'>16:40</var> PDT Monitoring - We have mitigated the issue and are monitoring the situation to ensure that provisioning is stable. Jul <var data-var='date'>15</var>, <var data-var='time'>16:35</var> PDT Investigating - We have identified that creating new VMs and clusters takes 10 mins or more. We are investigating the cause and will update this page when we have made some progress.

Compatibility Matrix clusters are taking longer than expected to start

Jun <var data-var='date'>29</var>, <var data-var='time'>19:29</var> PDT Resolved - The incident has been resolved Jun <var data-var='date'>29</var>, <var data-var='time'>14:47</var> PDT Identified - AKS instance types Standard_D2S_v5, Standard_D4S_v5, Standard_D8S_v5, Standard_D16S_v5, Standard_D32S_v5, and Standard_D48S_v5 are currently unavailable. We are working with Azure to correct this issue.

Some AKS instance types are unavailable

Jun <var data-var='date'>12</var>, <var data-var='time'>14:31</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>14:30</var> PDT Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>12</var>, <var data-var='time'>14:29</var> PDT Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>12</var>, <var data-var='time'>12:24</var> PDT Update - We are continuing to monitor upstream issues with Google Cloud Platform and Cloudflare Jun <var data-var='date'>12</var>, <var data-var='time'>11:23</var> PDT Monitoring - We are aware of upstream issues with Google Cloud Platform. This is presently impacting GKE components of our Compatibility Matrix product, and upstream GCR registries with our Replicated Registry. We are monitoring our cloud providers for updates.

High error rate from upstream cloud provider

Jun <var data-var='date'> 9</var>, <var data-var='time'>12:18</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>29</var>, <var data-var='time'>10:55</var> PDT Investigating - We're aware of network issues when creating an RKE2 cluster and are investigating

CMX: Network Issues when creating RKE2 clusters

Mar <var data-var='date'>31</var>, <var data-var='time'>20:09</var> PDT Resolved - This incident has been resolved. Mar <var data-var='date'>31</var>, <var data-var='time'>19:44</var> PDT Update - We are continuing to monitor for any further issues. Mar <var data-var='date'>31</var>, <var data-var='time'>19:43</var> PDT Monitoring - We have rolled out a fix, pinning GKE 1.32 to version 1.32.2, and are monitoring the results now. Mar <var data-var='date'>31</var>, <var data-var='time'>16:39</var> PDT Identified - We have identified an issue with pulling images from Google's registry, and are working with Google to find a resolution. Mar <var data-var='date'>31</var>, <var data-var='time'>15:31</var> PDT Investigating - We have noticed that GKE 1.32 clusters are not reaching a 'running' state. We are currently investigating this issue.

Is Replicated down?

Replicated status is UP

Lastest published events

Possible delays in provisioning CMX OpenShift relating to upstream dependencies

Issues rendering and loading docs.replicated.com

kurl.sh down

Replicated Services, Components and Regions

Recent Replicated events

Possible delays in provisioning CMX OpenShift relating to upstream dependencies

Issues rendering and loading docs.replicated.com

kurl.sh down

Issues with provisioning AKS in CMX

Issues with provisioning kURL in CMX

CMX Azure AKS is degraded

Get your FREE account now,
5 monitors included!

Is Replicated down?

Replicated status is UP

Lastest published events

Possible delays in provisioning CMX OpenShift relating to upstream dependencies

Issues rendering and loading docs.replicated.com

kurl.sh down

Replicated Services, Components and Regions

Recent Replicated events

Possible delays in provisioning CMX OpenShift relating to upstream dependencies

Issues rendering and loading docs.replicated.com

kurl.sh down

Issues with provisioning AKS in CMX

Issues with provisioning kURL in CMX

CMX Azure AKS is degraded

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!