Jun <var data-var='date'>24</var>, <var data-var='time'>06:55</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>24</var>, <var data-var='time'>03:29</var> PDT Monitoring - We have fixed the issue affecting our VMs and runs appear to be progressing normally now. There may still be some delays as we work through a small backlog of pending runs, and we will continue to monitor the situation. Jun <var data-var='date'>23</var>, <var data-var='time'>23:22</var> PDT Identified - We have discovered and issue with our Windows 11 VMs that is causing them to fail to boot and function. The fix has been identified and we are working to roll it out, but this may take a few hours. In the meantime runs that utilize these VM types will be delayed or may error. We are working to resolve the situation.

VM Capacity Issues

Nov <var data-var='date'>18</var>, <var data-var='time'>10:19</var> PST Resolved - This incident has been resolved. Nov <var data-var='date'>18</var>, <var data-var='time'>08:40</var> PST Update - Our provider is still reporting intermittent errors which may have an impact on some clients being able to access Rainforest services. We are continuing to monitor the situation. Nov <var data-var='date'>18</var>, <var data-var='time'>06:59</var> PST Monitoring - Our upstream provider has implemented a fix and services are returning to normal operation. We're continuing to monitor the situation to ensure everything has recovered. Nov <var data-var='date'>18</var>, <var data-var='time'>06:05</var> PST Update - Our upstream provider has identified the issue and is currently working on restoring service Nov <var data-var='date'>18</var>, <var data-var='time'>04:25</var> PST Update - Our upstream provider is aware of the issue and is continuing to investigate. Nov <var data-var='date'>18</var>, <var data-var='time'>03:51</var> PST Identified - We are currently experiencing an issue with one of our providers that is causing the application to be intermittently available.

Application Unavailable

Sep <var data-var='date'> 8</var>, <var data-var='time'>10:39</var> PDT Resolved - All systems are back to normal. Sep <var data-var='date'> 8</var>, <var data-var='time'>10:16</var> PDT Monitoring - We have implemented a fix and background job processing appears to be returning to normal. We will continue to investigate to ensure everything is working as it should be. There is a backlog of tests awaiting execution, so test results may continue to be delayed as we work our way through executing them all. Sep <var data-var='date'> 8</var>, <var data-var='time'>10:02</var> PDT Investigating - We are investigating an issue with our background job processing which is causing tasks to be delayed. This will impact starting new tests, execution of current tests, and many other pieces of functionality within the application. We are currently working to resolve this.

Background Job Processing is Delayed

Jul <var data-var='date'>28</var>, <var data-var='time'>06:16</var> PDT Resolved - We've addressed the issue and logins should be functional for all users. Jul <var data-var='date'>28</var>, <var data-var='time'>06:01</var> PDT Investigating - We're currently investigating an issue affecting user logins. Some users may be having trouble logging in.

Issues with logging in

Jun <var data-var='date'>12</var>, <var data-var='time'>14:13</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>13:47</var> PDT Monitoring - Our upstream provider appears to have addressed the issue and we are no longer seeing any issues, but will continue to monitor the situation. Jun <var data-var='date'>12</var>, <var data-var='time'>11:50</var> PDT Investigating - One of our upstream providers is currently experiencing an outage. We are monitoring the impact of this on our application.

Upstream Provider Outage

Apr <var data-var='date'>22</var>, <var data-var='time'>09:03</var> PDT Resolved - The fix has been fully applied and all AI services should be back to normal. Apr <var data-var='date'>22</var>, <var data-var='time'>08:39</var> PDT Monitoring - A fix has been applied and will be rolling out over the next hour. AI services should now be broadly available for most platforms but some older runs / previews may still be affected. We'll continue to monitor the situation until the rollout is complete. Apr <var data-var='date'>22</var>, <var data-var='time'>08:06</var> PDT Identified - We have identified an issue that is affecting step generation, self-healing snippets, and AI test assertions. Those service will be unavailable until such time as the incident is fixed.

Issues with self-healing and AI test assertions

Apr <var data-var='date'> 7</var>, <var data-var='time'>10:52</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'> 7</var>, <var data-var='time'>10:37</var> PDT Monitoring - We've implemented a fix and all delayed background jobs have been processed. We are monitoring to make sure everything continues working properly. Apr <var data-var='date'> 7</var>, <var data-var='time'>10:30</var> PDT Investigating - We are investigating an issue related to background job processing. Some results or runs may be delayed during this time.

Delays in job processing

Mar <var data-var='date'>28</var>, <var data-var='time'>09:48</var> PDT Resolved - This incident has been resolved. Mar <var data-var='date'>28</var>, <var data-var='time'>09:35</var> PDT Monitoring - The application is back online, and we're monitoring to make sure everything is working properly again. Mar <var data-var='date'>28</var>, <var data-var='time'>09:29</var> PDT Identified - The Rainforest application is currently offline. This affects access to the application, as well as any tests or runs that were in progress. We're looking into it and will provide an update shortly.

Rainforest service is unavailable

Jan <var data-var='date'> 7</var>, <var data-var='time'>06:01</var> PST Resolved - The backlog of runs has been processed and everything should be back to normal. Jan <var data-var='date'> 7</var>, <var data-var='time'>05:40</var> PST Monitoring - We've applied a fix and everything should be operational again. We're processing the backlog of runs created in the last 3 hours which will take some time. We'll update the status to "Resolved" once the backlog is processed. Jan <var data-var='date'> 7</var>, <var data-var='time'>04:33</var> PST Identified - We've identified an issue which is causing problems when trying to execute tests. Test runs are not being processed. Virtual machines in the Visual Editor are not available. We're working on a fix.

Issues with runs and test previews

Oct <var data-var='date'>10</var>, <var data-var='time'>10:52</var> PDT Resolved - We have resolved the issue affecting test video recording. Test runs from approximately 06:30 UTC to 17:35 UTC may be missing their video recordings. Oct <var data-var='date'>10</var>, <var data-var='time'>09:30</var> PDT Investigating - We are investigating an issue that is delaying the availability of test result videos

Delayed availability of test result videos

Oct <var data-var='date'> 2</var>, <var data-var='time'>13:15</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'> 2</var>, <var data-var='time'>12:28</var> PDT Monitoring - A permanent fix has been implemented and performance appears to have returned to normal. No further delays are expected. Oct <var data-var='date'> 2</var>, <var data-var='time'>06:42</var> PDT Update - We have applied a temporary fix to mitigate the issue and are continuing to work on a more permanent fix. Oct <var data-var='date'> 2</var>, <var data-var='time'>05:48</var> PDT Identified - We are working on an issue that is delaying the allocation of virtual machines when starting a run. We have identified the issue and are working to resolve it.

Delays when starting runs

Oct <var data-var='date'> 1</var>, <var data-var='time'>12:54</var> PDT Resolved - We've fixed the issue and runs are now starting and progressing as usual. Oct <var data-var='date'> 1</var>, <var data-var='time'>12:41</var> PDT Identified - We've identified an issue that may have caused some runs to be delayed.

Potential delays when kicking off runs

Sep <var data-var='date'>18</var>, <var data-var='time'>11:19</var> PDT Resolved - The incident has been resolved. Sep <var data-var='date'>18</var>, <var data-var='time'>10:52</var> PDT Monitoring - A fix has been implemented, we are now monitoring. Sep <var data-var='date'>18</var>, <var data-var='time'>09:43</var> PDT Investigating - We are currently investigating the issue.

Issues when setting up Rainforest Direct Connect tunnels

Jun <var data-var='date'>12</var>, <var data-var='time'>23:05</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>19:15</var> PDT Monitoring - We've shipped a fix and are monitoring the situation. Jun <var data-var='date'>12</var>, <var data-var='time'>17:02</var> PDT Update - We've identified the component we believe is causing the problem and are investigating why it's behaving badly. Jun <var data-var='date'>12</var>, <var data-var='time'>16:24</var> PDT Investigating - We've noticed that some test executions are not being processed correctly, which is holding up the completion of runs. We're currently investigating it.

Some runs not completing in a timely fashion

Jun <var data-var='date'> 7</var>, <var data-var='time'>07:46</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'> 7</var>, <var data-var='time'>05:57</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 7</var>, <var data-var='time'>05:11</var> PDT Investigating - Operations involving AI are taking some time to complete. We're investigating the issue.

Poor performance for AI related operations

Jun <var data-var='date'> 6</var>, <var data-var='time'>09:36</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'> 6</var>, <var data-var='time'>09:23</var> PDT Monitoring - We implemented a fix and are monitoring the results. Jun <var data-var='date'> 6</var>, <var data-var='time'>08:39</var> PDT Identified - We’ve identified another problem with action generation and are working on a fix

Issues with action generation

Jun <var data-var='date'> 6</var>, <var data-var='time'>08:07</var> PDT Resolved - The issue has been resolved Jun <var data-var='date'> 6</var>, <var data-var='time'>07:58</var> PDT Monitoring - We've shipped a fix and will continue to monitor the situation. Jun <var data-var='date'> 6</var>, <var data-var='time'>07:53</var> PDT Identified - We’ve identified a problem with action generation and are working on a fix”

May <var data-var='date'> 8</var>, <var data-var='time'>03:20</var> PDT Resolved - This incident has been resolved. May <var data-var='date'> 8</var>, <var data-var='time'>03:13</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'> 8</var>, <var data-var='time'>03:04</var> PDT Identified - We're experiencing issues with failure summarization. The fix is being deployed.

Automatic result analysis issues

Apr <var data-var='date'>15</var>, <var data-var='time'>11:24</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>15</var>, <var data-var='time'>05:49</var> PDT Monitoring - We've shipped a fix and will continue to monitor the situation. Apr <var data-var='date'>15</var>, <var data-var='time'>03:56</var> PDT Identified - We've identified the cause of the delays and are working on a fix. Apr <var data-var='date'>15</var>, <var data-var='time'>03:40</var> PDT Update - We are continuing to investigate this issue. Apr <var data-var='date'>15</var>, <var data-var='time'>03:38</var> PDT Investigating - We are currently experiencing an issue with runs being delayed and currently working on a fix.

Delayed Runs

Apr <var data-var='date'>12</var>, <var data-var='time'>11:23</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>12</var>, <var data-var='time'>09:59</var> PDT Update - The problem is somewhat improved, but we're continuing to investigate. Apr <var data-var='date'>12</var>, <var data-var='time'>07:59</var> PDT Investigating - We've noticed a problem with our fleet of VMs, which are used for test execution and test authoring. This may mean that you can't edit tests, and that test execution will be delayed.

VMs not available

Feb <var data-var='date'>26</var>, <var data-var='time'>21:23</var> PST Resolved - Confirmed that everything is back to normal. There should be no more delayed runs. Feb <var data-var='date'>26</var>, <var data-var='time'>18:55</var> PST Monitoring - We've resolved the issue and are continuing to monitor. Runs should be progressing normally again. Feb <var data-var='date'>26</var>, <var data-var='time'>17:36</var> PST Identified - We're experiencing some internal issues. Runs are being delayed. We've identified the source and are resolving it.

Runs are being delayed

Feb <var data-var='date'>23</var>, <var data-var='time'>09:29</var> PST Resolved - This incident has been resolved. Feb <var data-var='date'>23</var>, <var data-var='time'>08:30</var> PST Monitoring - Our new fix is applied and appears to be working properly. We will continue to monitor the situation to ensure it is fully resolved. Feb <var data-var='date'>23</var>, <var data-var='time'>06:39</var> PST Identified - There was an issue with the fix we had applied which caused the issue to re-appear. We've applied a new fix and things are starting to improve, but it may take a few minutes for the backlog to be processed. Any runs that were unable to proceed should begin progressing soon. Feb <var data-var='date'>23</var>, <var data-var='time'>05:22</var> PST Monitoring - We identified an issue that was preventing test editing and runs from proceeding. We've deployed a fix and are monitoring the situation.

Test environments not loading

Feb <var data-var='date'>12</var>, <var data-var='time'>08:13</var> PST Resolved - This incident has been resolved. Feb <var data-var='date'>12</var>, <var data-var='time'>08:11</var> PST Update - All Android tests are fully operational. Feb <var data-var='date'>12</var>, <var data-var='time'>03:22</var> PST Identified - We've identified the problem - it's restricted to tests and previews that use Android. We're working on a fix. Feb <var data-var='date'>12</var>, <var data-var='time'>00:01</var> PST Update - We are continuing to investigate this issue. Feb <var data-var='date'>12</var>, <var data-var='time'>00:00</var> PST Investigating - Our monitoring has identified an issue that may be causing some runs to be delayed. We're investigating the issue.

Runs delayed

Feb <var data-var='date'> 9</var>, <var data-var='time'>13:40</var> PST Resolved - This incident has been resolved. Any delayed runs are now in progress and Android VMs are available for use in the Visual Editor. Feb <var data-var='date'> 9</var>, <var data-var='time'>13:01</var> PST Investigating - We've detected a problem with Android VMs not loading in the Visual Editor. Runs may also be affected. We're investigating now.

Android VM Availability Issues

Nov <var data-var='date'> 9</var>, <var data-var='time'>05:22</var> PST Resolved - This incident has been resolved. Nov <var data-var='date'> 9</var>, <var data-var='time'>05:02</var> PST Monitoring - The application is back online. We are monitoring for any lingering performance degradation. Nov <var data-var='date'> 9</var>, <var data-var='time'>04:32</var> PST Investigating - The Rainforest app is currently experiencing a major outage. Engineers are actively working on a resolution.

Is Rainforest QA down?

Rainforest QA status is UP

Lastest published events

VM Capacity Issues

Application Unavailable

Background Job Processing is Delayed

Rainforest QA Services, Components and Regions

Recent Rainforest QA events

VM Capacity Issues

Application Unavailable

Background Job Processing is Delayed

Issues with logging in

Upstream Provider Outage

Issues with self-healing and AI test assertions