Apr <var data-var='date'> 6</var>, <var data-var='time'>23:00</var> UTC Completed - The scheduled maintenance has been completed.Apr <var data-var='date'> 6</var>, <var data-var='time'>21:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>29</var>, <var data-var='time'>23:15</var> UTC Scheduled - We will be undergoing maintenance on some shared servers at this time

Maintenance on shared servers

Apr <var data-var='date'> 2</var>, <var data-var='time'>15:12</var> UTC Resolved - The Management UI SSO shortcut is back to service. Impacted Clusters: All customers with clusters created more than 6 months ago Cause: Some code changes unexpectedly changed the communication schema between our backend services and the SSO component on the clusters. The proxy configuration conflicted with those changes causing the 502 Bad Gateway error to show up. Impact: Only the SSO via CloudAMQP Console was impacted given it required communication with our backend components. The AMQP Clusters were healthy and servicing traffic via both AMQP and HTTP on Management UI and API. Logging into the Management UI via Credentials and all Authentications Backends was functioning normally.Apr <var data-var='date'> 2</var>, <var data-var='time'>15:07</var> UTC Update - We are continuing to work on a fix for this issue.Apr <var data-var='date'> 2</var>, <var data-var='time'>14:52</var> UTC Update - The Management UI and brokers are serving traffic normally. You can still use the Management UI by using the Broker URL and properly typing User and Password. The impact is related to our backend service logs into Management UI via CloudAMQP SSO.Apr <var data-var='date'> 2</var>, <var data-var='time'>14:41</var> UTC Identified - Customers are experiencing Management UI being unavailable with 502 Bad Gateway. Timeline: - 14:16 UTC -> The issue was identified - 14:30 UTC -> Recent code changes were reverted. No relationship with incident noted.

Management Interface Unavailable - 502 Bad Gateway Response

Mar <var data-var='date'>25</var>, <var data-var='time'>18:03</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>25</var>, <var data-var='time'>17:45</var> UTC Identified - We're currently experiencing issues with metrics delivery, this affects 12% of the API based v1/v2 integrations. We've added additional capacity and expect have handled the backlog within 20 minutes.

Metrics delivery delay for 12% of the fleet

Mar <var data-var='date'>24</var>, <var data-var='time'>10:22</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>24</var>, <var data-var='time'>10:14</var> UTC Verifying - Verification is currently underway for the maintenance items.Mar <var data-var='date'>24</var>, <var data-var='time'>09:44</var> UTC Update - Metrics pipeline maintenance done. Currently doing maintenance on logs system.Mar <var data-var='date'>24</var>, <var data-var='time'>09:30</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>24</var>, <var data-var='time'>09:17</var> UTC Scheduled - Pipelines for metrics and log integrations are undergoing maintenance, we do not expect any significant delays.

Metrics and log integration maintenance

Mar <var data-var='date'>19</var>, <var data-var='time'>03:50</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>18</var>, <var data-var='time'>22:16</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'>18</var>, <var data-var='time'>21:25</var> UTC Identified - We noticed an issue with our queue metrics being sent to metrics integrations. there is currently a delay in processing, but we have identified an issue and expect the delay to be fixed shortly.

Queue metrics for legacy integrations delayed

Mar <var data-var='date'> 4</var>, <var data-var='time'>14:14</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'> 4</var>, <var data-var='time'>11:16</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'> 3</var>, <var data-var='time'>21:11</var> UTC Investigating - We're currently experiencing issues with Coralogix log delivery.

Coralogix log delivery performance issues

Mar <var data-var='date'> 3</var>, <var data-var='time'>20:51</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'> 3</var>, <var data-var='time'>19:30</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'> 3</var>, <var data-var='time'>18:47</var> UTC Identified - The issue has been identified and a fix is being implemented.Mar <var data-var='date'> 3</var>, <var data-var='time'>18:28</var> UTC Investigating - We're currently experiencing issues with metrics delivery, this affects v3 (Prometheus based) metrics.

Metrics delivery outage for v3 Prometheus based metrics.

Mar <var data-var='date'> 2</var>, <var data-var='time'>20:19</var> UTC Update - We are still experiencing issues with this region. If your services are impacted, we recommend you create a new cluster in a different region. When creating, use the copy config option, but do not copy plugin settings as this requires the source cluster to be running. Instead, enable once cluster is configured.Mar <var data-var='date'> 2</var>, <var data-var='time'>05:35</var> UTC Monitoring - AWS has an ongoing issue with multiple services in region me-south-1. Shared server hummingbird-01 affected.

AWS region me-south-1 outage

Mar <var data-var='date'> 2</var>, <var data-var='time'>20:19</var> UTC Update - We are still experiencing issues with all AZ's in this region. If your services are impacted, we recommend you create a new cluster in a different region. When creating, use the copy config option, but do not copy plugin settings as this requires the source cluster to be running. Instead, enable once cluster is configured.Mar <var data-var='date'> 2</var>, <var data-var='time'>01:46</var> UTC Update - We are continuing to monitor. The outage is in regards to the AZ mec1-az2Mar <var data-var='date'> 1</var>, <var data-var='time'>13:23</var> UTC Update - We are continuing to monitor for any further issues.Mar <var data-var='date'> 1</var>, <var data-var='time'>13:22</var> UTC Update - We are continuing to monitor for any further issues.Mar <var data-var='date'> 1</var>, <var data-var='time'>13:20</var> UTC Monitoring - AWS has an ongoing issue with multiple services in region me-central-1.

AWS region me-central-1 outage

Feb <var data-var='date'>18</var>, <var data-var='time'>15:30</var> UTC Resolved - This incident has been resolved.Feb <var data-var='date'>17</var>, <var data-var='time'>21:44</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Feb <var data-var='date'>17</var>, <var data-var='time'>19:31</var> UTC Investigating - We are currently investigating this issue.

Issues with shared server "chameleon"

Feb <var data-var='date'> 3</var>, <var data-var='time'>15:28</var> UTC Resolved - This incident has been resolved.Feb <var data-var='date'> 3</var>, <var data-var='time'>15:11</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Feb <var data-var='date'> 3</var>, <var data-var='time'>14:47</var> UTC Investigating - We are currently investigating this issue.

The shared server 'porpoise' is overloaded. We are investigating this!

Feb <var data-var='date'> 3</var>, <var data-var='time'>08:17</var> UTC Resolved - This incident has been resolved by Azure, https://azure.status.microsoft/en-us/statusFeb <var data-var='date'> 3</var>, <var data-var='time'>02:53</var> UTC Monitoring - A known Azure outage is leading to an error that prevents instances being created for now.

Azure outage

THIS IS A SCHEDULED EVENT Jan <var data-var='date'>31</var>, <var data-var='time'>01:20</var> UTC &nbsp;-&nbsp; Feb <var data-var='date'>23</var>, <var data-var='time'>09:20</var> UTCJan <var data-var='date'>30</var>, <var data-var='time'>20:26</var> UTC Scheduled - Shared RabbitMQ servers are being updated to version 4.2.x. If you are using JavaScript with amqplib/amqp-client.js, upgrade to version 10.7+/3.2.1+. RabbitMQ 4.1/2 introduces a new frame_max behavior that may cause compatibility issues.

Shared RabbitMQ servers upgrade

Jan <var data-var='date'>14</var>, <var data-var='time'>16:58</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'>14</var>, <var data-var='time'>15:56</var> UTC Monitoring - We are monitoring now.Jan <var data-var='date'>14</var>, <var data-var='time'>15:55</var> UTC Identified - The issue has been identified and a fix is being implementedJan <var data-var='date'>14</var>, <var data-var='time'>15:54</var> UTC Investigating - We are currently investigating this issue

Datadog metrics issue

Dec <var data-var='date'>11</var>, <var data-var='time'>13:12</var> UTC Resolved - This incident has been resolved by third party.Dec <var data-var='date'>11</var>, <var data-var='time'>12:16</var> UTC Investigating - emails and replies are delayed due to Helscout connection issues https://status.helpscout.com/incidents/kmd031b5fm16,

Email Delays

Dec <var data-var='date'>10</var>, <var data-var='time'>13:13</var> UTC Resolved - This incident has been resolved.Dec <var data-var='date'>10</var>, <var data-var='time'>10:13</var> UTC Investigating - Servers in west Europe can experience connection issues due to Virtual Machines issues at the provider. We will monitor and update when we know more.

Connectivity issues with Azure westeurope

Nov <var data-var='date'>19</var>, <var data-var='time'>16:03</var> UTC Resolved - This incident has been resolvedNov <var data-var='date'>19</var>, <var data-var='time'>15:49</var> UTC Update - We are continuing to work on a fix for this issue.Nov <var data-var='date'>19</var>, <var data-var='time'>15:19</var> UTC Identified - The issue has been identified and a fix is being implementedNov <var data-var='date'>19</var>, <var data-var='time'>15:07</var> UTC Investigating - We're currently experiencing issues with alarms notification/webhooks function.

Alarms/webhook notification issue

Nov <var data-var='date'>18</var>, <var data-var='time'>02:03</var> UTC Resolved - This incident has been resolved.Nov <var data-var='date'>17</var>, <var data-var='time'>22:57</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Nov <var data-var='date'>17</var>, <var data-var='time'>22:52</var> UTC Identified - Metrics are currently not showing for dedicated servers, this is related to an outage of one of our suppliers.

Metrics not showing

Nov <var data-var='date'> 6</var>, <var data-var='time'>01:50</var> UTC Resolved - We have observed that impacted clusters has restored services, and are working to ensure everything is back to normal.Nov <var data-var='date'> 5</var>, <var data-var='time'>19:21</var> UTC Monitoring - Update from Azure - Current Status: We detected this issue via our automated service monitoring, which identified a spike in hardware temperatures and service incidents affecting multiple clusters. We are actively investigating the cause and working with on-site technicians to restore normal cooling and service operations. The next update will be provided within 60 minutes, or as events warrant. AZ 1 and 2are impacted.Nov <var data-var='date'> 5</var>, <var data-var='time'>18:50</var> UTC Update - We are continuing to work on a fix for this issue.Nov <var data-var='date'> 5</var>, <var data-var='time'>17:47</var> UTC Identified - Access to Azure west Europe Availability zone 2 has been interrupted. Monitoring Azure and their report on the issue.

Azure westeurope Availability Zone: 2

Oct <var data-var='date'>30</var>, <var data-var='time'>15:30</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>30</var>, <var data-var='time'>03:31</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.

Issues with shared instance 'fuji'

Oct <var data-var='date'>29</var>, <var data-var='time'>17:58</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>29</var>, <var data-var='time'>17:24</var> UTC Identified - We've identified an issue in creating new clusters, when those clusters are using "copy-config" to get definitions from another cluster. Regular cluster creation still works fine.

Delay in creating new servers when using copy-config

Oct <var data-var='date'>24</var>, <var data-var='time'>05:38</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>24</var>, <var data-var='time'>02:21</var> UTC Investigating - We are currently investigating. Creating shared accounts work, but not dedicated. Don't trigger upgrades or plan changes if it can be avoided.

We are having issues creating and upgrading clusters

Oct <var data-var='date'>20</var>, <var data-var='time'>20:39</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>20</var>, <var data-var='time'>19:32</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Oct <var data-var='date'>20</var>, <var data-var='time'>17:14</var> UTC Identified - Our service provider for support system is partially down. We can still reach many parts of the system and critical incidents still trigger incident response. If you have issues getting to CloudAMQP support you can use our backup email address support-group@84codes.com

CloudAMQP Support System partial outage

Oct <var data-var='date'>13</var>, <var data-var='time'>20:27</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>13</var>, <var data-var='time'>17:51</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Oct <var data-var='date'>13</var>, <var data-var='time'>17:01</var> UTC Identified - We're investigating issues with delayed or missing metrics

Some metrics are not delivered correctly

Oct <var data-var='date'>13</var>, <var data-var='time'>17:00</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>13</var>, <var data-var='time'>16:56</var> UTC Monitoring - We've found the issues and accounts are now being processed as they should.Oct <var data-var='date'>13</var>, <var data-var='time'>16:44</var> UTC Identified - The issue has been identified and a fix is being implemented.

Is CloudAMPQ down?

CloudAMPQ status is PARTIAL OUTAGE

Components with issues

Lastest published events

Maintenance on shared servers

Management Interface Unavailable - 502 Bad Gateway Response

Metrics delivery delay for 12% of the fleet

CloudAMPQ Services, Components and Regions

Recent CloudAMPQ events

Maintenance on shared servers

Management Interface Unavailable - 502 Bad Gateway Response

Metrics delivery delay for 12% of the fleet

Metrics and log integration maintenance

Queue metrics for legacy integrations delayed

Coralogix log delivery performance issues

Get your FREE account now,
5 monitors included!

Is CloudAMPQ down?

CloudAMPQ status is PARTIAL OUTAGE

Components with issues

Lastest published events

Maintenance on shared servers

Management Interface Unavailable - 502 Bad Gateway Response

Metrics delivery delay for 12% of the fleet

CloudAMPQ Services, Components and Regions

Recent CloudAMPQ events

Maintenance on shared servers

Management Interface Unavailable - 502 Bad Gateway Response

Metrics delivery delay for 12% of the fleet

Metrics and log integration maintenance

Queue metrics for legacy integrations delayed

Coralogix log delivery performance issues

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!