Aug <var data-var='date'>19</var>, <var data-var='time'>14:00</var> UTC Resolved - The issue affecting TestRail has been resolved and all services are now fully operational. Thank you for your patience. Aug <var data-var='date'>19</var>, <var data-var='time'>12:34</var> UTC Monitoring - A fix has been implemented and we are monitoring the recovery. Some temporary instability may occur during this period as the system recovers. We will update once the system is fully stable. Aug <var data-var='date'>19</var>, <var data-var='time'>11:32</var> UTC Investigating - We are aware of an issue affecting TestRail where some users may experience slower performance and/or 500 errors. Our team is working to restore full functionality. Updates will follow as soon as we have more to share.

Performance issue impacting access to TestRail

Aug <var data-var='date'>17</var>, <var data-var='time'>12:00</var> UTC Resolved - We identified a DNS issue that may have prevented some users from accessing TestRail. The issue has been resolved, and all services are now fully operational. We apologize for any inconvenience and thank you for your patience.

Resolved: DNS Issue Affecting Access to TestRail

Aug <var data-var='date'>14</var>, <var data-var='time'>15:39</var> UTC Resolved - The issue affecting a small subset of customers has been resolved and all services are now fully operational. Thank you for your patience. Aug <var data-var='date'>12</var>, <var data-var='time'>16:15</var> UTC Monitoring - We continue to closely monitor performance for the affected Cloud instances following the mitigation applied earlier. Our team has identified a permanent resolution, which is currently undergoing review and testing. Thank you for your continued patience while we finalize this work. Aug <var data-var='date'>11</var>, <var data-var='time'>18:03</var> UTC Identified - We identified a performance issue affecting a subset of TestRail Cloud accounts, resulting in slower page load times for some users. Our team identified the underlying cause and implemented a mitigation that has stabilized performance. We are continuing to closely monitor the environment while we work toward a permanent resolution. We appreciate your patience and understanding as we work to fully resolve the issue.

Performance Degradation Affecting a Subset of Cloud Instances

Mar <var data-var='date'>23</var>, <var data-var='time'>22:20</var> UTC Resolved - Following the implementation of the fix, we have completed a period of monitoring and have confirmed that the issue affecting authentication with the Jira integration has been fully resolved, with no further errors observed. Users should ensure they have reauthenticated with Jira within TestRail to maintain normal functionality. Thank you for your patience while we worked to resolve this issue. Mar <var data-var='date'>22</var>, <var data-var='time'>20:06</var> UTC Monitoring - A fix has been implemented for the issue affecting authentication with the Jira integration. We are currently monitoring to ensure the issue is fully resolved. Users will need to reauthenticate with Jira within TestRail for the fix to take effect. After reauthenticating, normal functionality should resume. We will continue to monitor and provide further updates as needed. Mar <var data-var='date'>22</var>, <var data-var='time'>00:27</var> UTC Update - We are continuing to make progress on implementing the fix. Some customers may still experience issues, including “500 server error” messages when using the Jira integration. We appreciate your patience and will share further updates as they become available. Mar <var data-var='date'>20</var>, <var data-var='time'>22:27</var> UTC Update - Work to implement a fix is ongoing. During this time, some users may encounter “500 server error” messages when attempting to authenticate with or use the Jira integration. We will share additional updates as they become available. Mar <var data-var='date'>20</var>, <var data-var='time'>17:56</var> UTC Update - We are continuing to implement a fix for the issue. Further updates will be provided as progress continues. Mar <var data-var='date'>20</var>, <var data-var='time'>15:31</var> UTC Identified - We have identified the root cause of the issue affecting authentication for the Jira integration and are actively working to implement a fix. Some users may continue to experience failures when connecting to Jira or syncing data between TestRail and Jira during this time. We will provide further updates as remediation progresses. Mar <var data-var='date'>20</var>, <var data-var='time'>15:30</var> UTC Update - We are continuing to investigate this issue. Mar <var data-var='date'>20</var>, <var data-var='time'>02:00</var> UTC Investigating - We are currently investigating an issue affecting authentication for the Jira integration. Some users may experience failures when connecting to Jira or syncing data between TestRail and Jira. Our team is actively working to identify the root cause and will provide updates as more information becomes available.

Jira Integration Degradation

Feb <var data-var='date'>27</var>, <var data-var='time'>20:02</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>27</var>, <var data-var='time'>15:50</var> UTC Update - We're pleased to report significant performance improvements across our platform, with decreased error rates and API rate limits restored to their normal levels. Our team continues to closely monitor the system to ensure ongoing stability and reliability. Thank you for your patience and support. Feb <var data-var='date'>27</var>, <var data-var='time'>09:02</var> UTC Update - Some customers are experiencing degraded performance. Our engineers are investigating. Feb <var data-var='date'>27</var>, <var data-var='time'>00:10</var> UTC Monitoring - The incident has been resolved, and all customers should now have access to their instances. Please note that we have temporarily increased the API rate limit restrictions, and reduced the background task frequency while we monitor the system. We appreciate your patience during this time, and are committed to maintaining service stability. Feb <var data-var='date'>26</var>, <var data-var='time'>20:19</var> UTC Update - Some customers may experience maintenance windows longer than 15 minutes as our team continues to work to resolve the incident. Feb <var data-var='date'>26</var>, <var data-var='time'>18:57</var> UTC Identified - Our team is actively working to resolve the incident. Customers may see their instances in maintenance mode intermittently, but for no longer than 15 minutes. Feb <var data-var='date'>26</var>, <var data-var='time'>16:22</var> UTC Investigating - Some customers may be experiencing degraded performance. Our engineers are investigating.

Degraded Performance

Feb <var data-var='date'>19</var>, <var data-var='time'>13:44</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>19</var>, <var data-var='time'>12:34</var> UTC Identified - Our engineers have identified the potential source and implemented a fix. Feb <var data-var='date'>19</var>, <var data-var='time'>10:48</var> UTC Investigating - Some customers may be experiencing degraded performance. Our engineers are investigating.

Feb <var data-var='date'>18</var>, <var data-var='time'>23:48</var> UTC Resolved - We successfully resolved the issue, and normal service has been restored. Feb <var data-var='date'>16</var>, <var data-var='time'>15:53</var> UTC Update - We're continuing to monitor for any further issues while the team is working to a full resolution. Feb <var data-var='date'>15</var>, <var data-var='time'>15:32</var> UTC Monitoring - We identified degraded performance that may affect some customers in connection with the 14-Feb partial outage. Our team is actively investigating to ensure a complete resolution.

Degraded Performance for Some Customers

Feb <var data-var='date'>15</var>, <var data-var='time'>08:35</var> UTC Resolved - The incident has been resolved. We are continuing to monitor for errors. Feb <var data-var='date'>14</var>, <var data-var='time'>18:08</var> UTC Update - Monitoring continues while the rate of errors continues to reduce sharply. Feb <var data-var='date'>14</var>, <var data-var='time'>12:30</var> UTC Update - We are continuing to monitor this issue. Feb <var data-var='date'>14</var>, <var data-var='time'>10:59</var> UTC Monitoring - Customers should be seeing improvements. We will continue to monitor whilst services return to normal. Feb <var data-var='date'>14</var>, <var data-var='time'>09:14</var> UTC Identified - We're currently experiencing a partial Cloud outage which may result in error messages relating to failing API calls or attachment related nodes. We're working on the issue right now and expect to have a resolution shortly.

Partial outage due to an overrunning maintenance task

Oct <var data-var='date'>16</var>, <var data-var='time'>21:24</var> UTC Resolved - Our DevOps team has applied a fix for this issue and after a period of monitoring, has determined that users should no longer be experiencing issues related to the TestRail attachments service. We appreciate your patience while we dealt with this matter. If you're still experiencing errors that result in a "Call to a member function prepare()" message, please reach out to our support team and we'll be happy to assist. Oct <var data-var='date'>16</var>, <var data-var='time'>07:08</var> UTC Monitoring - Our team is stabilising our hosted infrastructure after applying the necessary fixes. We are continuing to monitor the situation closely. Oct <var data-var='date'>14</var>, <var data-var='time'>08:45</var> UTC Update - We are continuing to work on a fix for this issue which is affecting a small number of customers. Thanks for your patience. Our team is working hard to complete this activity as quickly as possible and we appreciate your bearing with us while we do so. Oct <var data-var='date'>13</var>, <var data-var='time'>08:24</var> UTC Update - We're still working to fix this issue as fast as we can. Thanks for your patience while we do so. We'll share updates as progress is made. Oct <var data-var='date'>12</var>, <var data-var='time'>17:20</var> UTC Identified - The issue has been identified and a fix is being implemented. Oct <var data-var='date'>12</var>, <var data-var='time'>17:20</var> UTC Investigating - The TestRail Cloud Attachments service is currently experiencing performance issues affecting a subset of attachments functionality for a small number of TestRail Cloud users. Our DevOps team is deploying a fix and we expect to resolve this matter promptly.

Degraded Cloud Attachments Service

Jun <var data-var='date'> 1</var>, <var data-var='time'>11:43</var> UTC Resolved - All systems are operational. Jun <var data-var='date'> 1</var>, <var data-var='time'>09:30</var> UTC Monitoring - The database has been recovered and all affected TestRail cloud instances should now be back online. We are continuing to monitor the database for any further issues. Jun <var data-var='date'> 1</var>, <var data-var='time'>08:28</var> UTC Identified - The cause of the database issue has been identified and our engineers are working to solve the problem. In the meantime, we are also building a new database from a backup with a view towards switching affected customers to the new database. Jun <var data-var='date'> 1</var>, <var data-var='time'>07:35</var> UTC Investigating - One of our databases is not responding correctly and as a result, TestRail instances reliant on it are currently in maintenance mode. Our DevOps team is investigating the problem.

Some TestRail instances are currently in maintenance mode

Sep <var data-var='date'> 7</var>, <var data-var='time'>13:17</var> UTC Resolved - The issue has been identified and a fix implemented. We will be continuing to monitor the situation. Sep <var data-var='date'> 7</var>, <var data-var='time'>12:08</var> UTC Investigating - We are currently investigating this issue.

A small number of customers are experiencing slow response times

Sep <var data-var='date'> 5</var>, <var data-var='time'>14:03</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'> 5</var>, <var data-var='time'>12:57</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'> 5</var>, <var data-var='time'>09:28</var> UTC Identified - We have identified the cause of this issue and we are currently in the process of implementing a fix. Sep <var data-var='date'> 5</var>, <var data-var='time'>08:53</var> UTC Investigating - We are currently investigating this issue.

Some customers are experiencing 403 Access Denied messages

Aug <var data-var='date'>18</var>, <var data-var='time'>17:58</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>17</var>, <var data-var='time'>17:24</var> UTC Monitoring - We have fixed the issue encountered by some customers earlier today in regards to Gateway Access issues. We are monitoring the situation. Aug <var data-var='date'>17</var>, <var data-var='time'>16:46</var> UTC Identified - We have identified the issue in which some customers have experienced gateway errors (502 - Bad Gateway, 504 - Gateway Time-out, 403 - Access Denied) when attempting to access their TestRail Cloud instance. We have resolved the issue for several of our customer instances and expect to resolve the remaining instances shortly. Aug <var data-var='date'>17</var>, <var data-var='time'>16:46</var> UTC Update - We are continuing to investigate this issue. Aug <var data-var='date'>17</var>, <var data-var='time'>16:45</var> UTC Update - We are continuing to investigate this issue. Aug <var data-var='date'>17</var>, <var data-var='time'>12:22</var> UTC Investigating - Some customers are experiencing gateway errors, e.g. '502 Bad Gateway', '504 Gateway Time-out', or '403-Access Denied' when attempting to access their TestRail cloud instance. We are currently investigating this issue and will provide further updates shortly.

Gateway and Access Denied errors when accessing TestRail

Jun <var data-var='date'>22</var>, <var data-var='time'>12:15</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>14</var>, <var data-var='time'>14:19</var> UTC Monitoring - We have identified the cause of the instability and implemented a fix which we are currently monitoring. Jun <var data-var='date'>14</var>, <var data-var='time'>12:58</var> UTC Investigating - Some customers are experiencing slower than expected TestRail performance. We are currently investigating this issue.

Slow performance

May <var data-var='date'> 4</var>, <var data-var='time'>14:35</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 4</var>, <var data-var='time'>10:46</var> UTC Monitoring - We have deployed the fix to all customers and are monitoring for any further issues. May <var data-var='date'> 4</var>, <var data-var='time'>09:24</var> UTC Update - Our team has identified the issue and implemented a fix. All cloud customers will receive the fix shortly. May <var data-var='date'> 4</var>, <var data-var='time'>08:24</var> UTC Update - We are continuing to investigate this issue and will provide further updates shortly. In the meantime, users are able to workaround the problem by using the "Pass & Next" button to add a result instead of the "Add Result" button. May <var data-var='date'> 4</var>, <var data-var='time'>07:18</var> UTC Investigating - Some customers are experiencing an error when attempting to add test results. We are currently investigating the cause of this issue and will provide an update shortly.

Undefined Column Name Bucket

Mar <var data-var='date'> 8</var>, <var data-var='time'>16:57</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 8</var>, <var data-var='time'>14:38</var> UTC Investigating - Some customers are experiencing gateway errors, e.g. '502 Bad Gateway' or '504 Gateway Time-out', when attempting to access their TestRail cloud instance. We are currently investigating this issue and will provide further updates shortly.

Gateway errors when accessing TestRail

Mar <var data-var='date'> 8</var>, <var data-var='time'>10:59</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 7</var>, <var data-var='time'>14:00</var> UTC Monitoring - Earlier today we had some further reports of further 504 and 502 gateway errors affecting a small number of customers. We've identified the cause of the errors and implemented a fix. Service to TestRail Cloud has been fully restored and all instances are showing as online and operational. We will continue to monitor closely to ensure there are no other issues. If you are still receiving any 502 or 504 gateway errors, please email us at contact@gurock.com so that we can take a closer look at your specific instance. Mar <var data-var='date'> 7</var>, <var data-var='time'>12:58</var> UTC Investigating - Some customers are experiencing gateway errors, e.g. '502 Bad Gateway' or '504 Gateway Time-out', when attempting to access their TestRail cloud instance. We are currently investigating this issue and will provide further updates shortly.

Mar <var data-var='date'> 7</var>, <var data-var='time'>11:05</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 7</var>, <var data-var='time'>09:26</var> UTC Monitoring - We have identified and resolved the issue. All instances are operational, however we are continuing to closely monitor the situation. Mar <var data-var='date'> 7</var>, <var data-var='time'>09:08</var> UTC Investigating - TestRail Cloud is currently experiencing issues. We know this is frustrating and we’re working to resolve this as soon as possible. We have escalated this to our engineering team, and will provide an update as soon as more information becomes available.

TestRail Cloud instances in Maintenance mode

Mar <var data-var='date'> 3</var>, <var data-var='time'>11:00</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 2</var>, <var data-var='time'>09:08</var> UTC Update - Earlier today we had some further reports of further 504 and 502 gateway errors affecting a small number of customers. We've identified the cause of the errors and implemented a fix. Service to TestRail Cloud has been fully restored and all instances are showing as online and operational. We will continue to monitor closely to ensure there are no other issues. If you are still receiving any 502 or 504 gateway errors, please email us at contact@gurock.com so that we can take a closer look at your specific instance. Mar <var data-var='date'> 2</var>, <var data-var='time'>01:44</var> UTC Monitoring - TestRail Cloud is fully operational, but we are continuing to monitor and investigate reports of intermittent 504 and 502 gateway errors, including for the TestRail API, as well as some other issues related to timeouts. We will post an update as soon as we have more information. If you or a teammate is experiencing any issues, please notify our Support team at contact@gurock.com so we can help triage.

Intermittent gateway issues / performance degradation

Mar <var data-var='date'> 1</var>, <var data-var='time'>20:10</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 1</var>, <var data-var='time'>20:09</var> UTC Monitoring - We've identified the cause of the gateway errors and implemented a fix. Service to TestRail Cloud has been fully restored and all instances are showing as online and operational. We will continue to monitor closely to ensure there are no other issues. If you are still receiving any 502 or 504 gateway errors, please email us at contact@gurock.com so that we can take a closer look at your specific instance. Mar <var data-var='date'> 1</var>, <var data-var='time'>18:35</var> UTC Update - TestRail Cloud continues to experience issues with 502 or 504 gateway errors. We know this is frustrating and we’re working to resolve this as soon as possible. We have escalated this to our engineering team, and will provide an update as soon as more information becomes available. Mar <var data-var='date'> 1</var>, <var data-var='time'>13:39</var> UTC Update - We are continuing to investigate this issue. Mar <var data-var='date'> 1</var>, <var data-var='time'>11:23</var> UTC Update - We are continuing to investigate this issue. We do not have an ETA for resolution at the present time, but expect to provide one shortly. Mar <var data-var='date'> 1</var>, <var data-var='time'>09:11</var> UTC Investigating - Some customers are experiencing gateway errors, e.g. '502 Bad Gateway' or '504 Gateway Time-out', when attempting to access their TestRail cloud instance. We are currently investigating this issue and will provide further updates shortly.

Jan <var data-var='date'>20</var>, <var data-var='time'>19:25</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>20</var>, <var data-var='time'>19:14</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jan <var data-var='date'>20</var>, <var data-var='time'>18:36</var> UTC Identified - The issue has been identified and a fix is being implemented. Jan <var data-var='date'>20</var>, <var data-var='time'>18:19</var> UTC Investigating - We are actively investigating an issue that may be causing slow performance or issues accessing certain TestRail Cloud instances.

Investigating some reports of degraded performance

Dec <var data-var='date'>29</var>, <var data-var='time'>21:00</var> UTC Resolved - Multiple Instances were stuck in maintenance mode. We have quickly identified and resolved the issue after receiving the reports.

Instances in Maintenance Mode

Nov <var data-var='date'>19</var>, <var data-var='time'>10:00</var> UTC Resolved - Users of TestRail were seeing ‘Cannot achieve consistency level LOCAL_ONE’ and ‘Request timed out’ errors. The issue was related to overloaded Cassandra nodes, but has now been resolved.

Degraded attachment performance

Nov <var data-var='date'> 8</var>, <var data-var='time'>17:06</var> UTC Resolved - We are working directly with affected customers and/or customer instances and will be restoring data to them over the next ~2 weeks. We will contact affected customers directly to confirm when this has been done. Since instances are functioning correctly at this time, we are treating this incident as resolved and no further updates will be made here. Oct <var data-var='date'> 4</var>, <var data-var='time'>23:09</var> UTC Update - We have successfully recovered data that was added to affected instances between September 28, 2021, 1:11:11 AM UTC and September 28, 2021, 12:12PM UTC. We will reach out to provide more details about how we will be restoring recovered data to affected TestRail instances once we have more information. Please note that you are still able to use your instance at this time, as we will be working to merge your current data with the restored data. Oct <var data-var='date'> 1</var>, <var data-var='time'>18:58</var> UTC Update - There are no new updates at this time. We are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. Please note that you are still able to use your instance at this time, as we will be working to merge the current data after the restore has completed. We will post another update on October 4, 2021, at 15:00 UTC, unless we receive more information before that. Sep <var data-var='date'>30</var>, <var data-var='time'>23:04</var> UTC Update - There are no new updates at this time. We are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. We will post another update at or before October 1, 2021, at 15:00 UTC, unless we receive more information before that. Sep <var data-var='date'>30</var>, <var data-var='time'>15:06</var> UTC Update - All Cloud systems are operational, but we are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. We will post another update at or before September 30, 2021, (today) at 22:30 UTC, unless we receive more information before that. Sep <var data-var='date'>29</var>, <var data-var='time'>14:16</var> UTC Update - Per our earlier update, all affected instances should now be accessible. It’s possible that affected instances may still be experiencing challenges with lost data however. This is due to the fact that the instances have been restored from a backup prior to the crash. The backup used has the following timestamp: September 28, 2021, 1:11:11 AM UTC. What this means: Data added to your instance after the timestamp but before the outage may not have been fully restored to your instance. We are continuing to work with AWS to recover and restore data added to your instance after the backup timestamp, but this process is expected to take a little while. We will update this status once we have completed our work with AWS to restore all of your data. We appreciate your patience and apologise for any inconvenience caused in the meantime. Sep <var data-var='date'>29</var>, <var data-var='time'>10:15</var> UTC Monitoring - All affected instances are now functioning. Data has been restored to the affected instances from a backup prior to the crash: September 28, 2021, 1:11:11 AM UTC. We are working to restore any changes after this time to the affected instances, and will continue to provide updates on progress as we make it. Sep <var data-var='date'>29</var>, <var data-var='time'>07:34</var> UTC Update - We’re continuing to work with AWS Support and we will keep you updated as the situation progresses. Sep <var data-var='date'>28</var>, <var data-var='time'>17:03</var> UTC Update - We are continuing to implement the fix identified earlier today, and service has been restored to a subset of affected instances. We will provide an update as soon as we have more information. Sep <var data-var='date'>28</var>, <var data-var='time'>13:32</var> UTC Identified - The issue has been identified and a fix is being implemented. Sep <var data-var='date'>28</var>, <var data-var='time'>12:56</var> UTC Investigating - We are currently investigating this issue.

Some instances are currently showing maintenance mode.

Oct <var data-var='date'>21</var>, <var data-var='time'>09:19</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>21</var>, <var data-var='time'>09:13</var> UTC Investigating - We are currently investigating this issue.

Is TestRail Cloud down?

TestRail Cloud status is UP

Lastest published events

Performance issue impacting access to TestRail

Resolved: DNS Issue Affecting Access to TestRail

Performance Degradation Affecting a Subset of Cloud Instances

TestRail Cloud Services, Components and Regions

Recent TestRail Cloud events

Performance issue impacting access to TestRail

Resolved: DNS Issue Affecting Access to TestRail

Performance Degradation Affecting a Subset of Cloud Instances

Jira Integration Degradation

Degraded Performance

Degraded Performance