Jul <var data-var='date'>25</var>, <var data-var='time'>13:49</var> UTC Resolved - All planned updates have been completed. Jul <var data-var='date'>25</var>, <var data-var='time'>09:22</var> UTC Investigating - We are performing database updates in the Robocorp automation platform today. This may result in minor disruptions in service. Practical impact: 1. Identity infrastructure: SSO based logins will be unavailable for a total of 20-30 minutes in each region. This also affects token refreshes, resulting in a chance of logged in users being logged out during the window. Running automations are not affected by this downtime. 2. Execution infrastructure: short window (< 1 minute) of disruptions for each region. The core infrastructure is resilient and we expect no adverse impact to running and scheduled automations, but you may experience e.g. increased rate of failed API requests. The upgrades will be performed within the next few hours. We will close this incident once complete.

Maintenance day

Nov <var data-var='date'>18</var>, <var data-var='time'>17:09</var> UTC Resolved - Although Cloudflare still has open issues, our site has been back up and stable for a while. We will close this issue but continue monitoring the situation. Nov <var data-var='date'>18</var>, <var data-var='time'>14:11</var> UTC Investigating - Sema4.ai corporate website at https://sema4.ai and documentation at https://sema4.ai/docs are impacted by the ongoing Cloudflare outage. Please see https://www.cloudflarestatus.com/ for the most up-to-date status updates. We expect the issues to resolve once Cloudflare restores service.

Website and documentation impacted by Cloudflare issues

Oct <var data-var='date'>21</var>, <var data-var='time'>07:05</var> UTC Resolved - The issue affecting our Control Room products in the U.S. region has been fully resolved. All services are now operating normally. Please note that scheduled runs and API triggers initiated during the outage period—between 11:40 PM PDT on October 19 and 3:55 PM PDT on October 20—were likely impacted and may not have executed. We recommend reviewing the status of affected runs in Control Room and manually re-running any missed processes as needed. Oct <var data-var='date'>20</var>, <var data-var='time'>12:36</var> UTC Monitoring - We're monitoring issues on our US based deployments. Some of the services are recovering but at the moment there are issues with e.g. robot process run executions and scheduling.

Issues on US based deployments

Mar <var data-var='date'>15</var>, <var data-var='time'>08:55</var> UTC Resolved - Upgrade is complete and we are monitoring the performance. Mar <var data-var='date'>15</var>, <var data-var='time'>08:28</var> UTC Investigating - Today's database upgrade 3/3: SSO integration database engine upgrade for US based deployments is in progress. SSO-based users may encounter failures in login or session refresh. Backend services or API access is not impacted.

Database engine upgrade on SSO authentication database, US based deployments

Mar <var data-var='date'>15</var>, <var data-var='time'>08:26</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>15</var>, <var data-var='time'>08:26</var> UTC Update - Upgrade is complete and we are monitoring the performance. Mar <var data-var='date'>15</var>, <var data-var='time'>08:00</var> UTC Identified - Today's database upgrade 2/3: SSO integration database engine upgrade for EU based deployments is in progress. SSO-based users may encounter failures in login or session refresh. Backend services or API access is not impacted.

Database engine upgrade on SSO authentication database, EU based deployments

Mar <var data-var='date'>15</var>, <var data-var='time'>07:54</var> UTC Resolved - Upgrade is complete and we are monitoring the performance. Mar <var data-var='date'>15</var>, <var data-var='time'>07:35</var> UTC Investigating - Today's database upgrade 1/3: core database engine upgrade for eu1 (Ireland) Control Room instance is in progress. We expect slightly increased error rates during a 10 minute time window in all Control Room services.

Database engine upgrade on eu1 (Ireland) Control Room instance

Feb <var data-var='date'>25</var>, <var data-var='time'>10:23</var> UTC Resolved - Upgrade is complete. We apologize for any inconvenience caused by the maintenance. Feb <var data-var='date'>25</var>, <var data-var='time'>10:07</var> UTC Investigating - As previously announced, we are performing a database update on the us1 Control Room instance hosted in N. Virginia. Occasional API call failures or other minor disruptions in service may be expected during the update. Our team is closely monitoring the progress, and we will keep this page up to date with status.

Database engine update on Control Room us1 (AWS us-east-1) instance

Feb <var data-var='date'>25</var>, <var data-var='time'>08:30</var> UTC Resolved - Upgrade on eu2 is complete. Apologies for the inconvenience. Feb <var data-var='date'>25</var>, <var data-var='time'>08:16</var> UTC Investigating - As previously announced, we are performing a database update on the eu2 Control Room instance hosted in Frankfurt, Germany. Occasional API call failures or other minor disruptions in service may be expected during the update. Our team is closely monitoring the progress, and we will keep this page up to date with status.

Database engine update on Control Room eu2 (AWS eu-central-1) instance

Aug <var data-var='date'> 5</var>, <var data-var='time'>09:23</var> UTC Resolved - This issue has been resolved. The issue impacted only end-user facing feature. Any automations running on the platform were not impacted. Our apologies for the inconvenience. Aug <var data-var='date'> 5</var>, <var data-var='time'>09:13</var> UTC Investigating - We are investigating issues with failure to log in to Control Room.

Control Room login failures

Jan <var data-var='date'>19</var>, <var data-var='time'>10:00</var> UTC Resolved - We encountered increased error rates in database access during a ~10-minute window around noon today on the eu1 self-service instance accessible at https://cloud.robocorp.com. A small number of runs may be affected. The situation is back to normal. We apologize for the inconvenience. If you have any questions, please get in touch with us.

Increased error rates on eu1 self-service instance

Jul <var data-var='date'>25</var>, <var data-var='time'>19:32</var> UTC Resolved - The issue has been resolved and we have deployed additional measures to avoid this problem in the future. Some data may still be out of sync in the indexes, and we continue on re-syncing. Jul <var data-var='date'>25</var>, <var data-var='time'>15:50</var> UTC Monitoring - The situation appears to be back to normal. We are still monitoring the situation and starting work on measures to ensure this does not happen in the future. Jul <var data-var='date'>25</var>, <var data-var='time'>15:12</var> UTC Investigating - We are currently observing increased error rates in indexing service of eu1 Control Room (https://cloud.robocorp.com) and our team is investigating the situation as a top priority. The indexing service powers various listing and search features, but is not a system of record for any data or part of the operational path of robots. In practice, there may be discrepancies between the actual data and data shown in various searchable listings.

Increased error rates in indexing service

Jul <var data-var='date'>24</var>, <var data-var='time'>16:31</var> UTC Resolved - Situation is back to normal. Subset of messages in our indexing service failed to process in a time window between 14:25 - 15:25 UTC today. The indexing service powers various listing and search features, but is not a system of record for any data or part of the operational path of robots. In practice, there may be discrepancies between the actual data and data shown in various searchable listings. We will continue to analyze the impact and are considering either replaying the messages or contacting impacted customers, depending on the findings. Jul <var data-var='date'>24</var>, <var data-var='time'>15:10</var> UTC Investigating - We are currently observing increased error rates in indexing service of eu1 Control Room (https://cloud.robocorp.com) and our team is investigating the situation.

Jul <var data-var='date'>20</var>, <var data-var='time'>22:13</var> UTC Resolved - This issue has been fixed. Please note that the issue impacted only serving the videos. Any run videos recorded earlier are now available for viewing. Jul <var data-var='date'>20</var>, <var data-var='time'>11:17</var> UTC Identified - There is a known issue with the run video feed view failing. Only the viewing infrastructure is experiencing problems. Recording is functioning correctly and recorded videos will be available once the issue is addressed. We aim to get a fix rolled by end of the day today.

Video feed view failing

Jun <var data-var='date'>13</var>, <var data-var='time'>21:18</var> UTC Resolved - Most AWS services have recovered and Control Room is returning to normal operation. We continue monitoring the situation. Jun <var data-var='date'>13</var>, <var data-var='time'>20:46</var> UTC Identified - The issue is confirmed to be caused by a broader problem on AWS services in us-east-1 region. The AWS team is working on getting things back to normal and we are already seeing significant improvements in the performance. Currently Control Room is operational with occasional hiccups. Jun <var data-var='date'>13</var>, <var data-var='time'>19:18</var> UTC Investigating - We are investigating an issue affecting Control Room services in AWS us-east-1 region. The Service is severely impacted by increased error rates. This incident appears to be connected with an operational issue on AWS services as reported on https://health.aws.amazon.com/health/status.

US instance service degradation

May <var data-var='date'>18</var>, <var data-var='time'>11:08</var> UTC Resolved - This incident has been resolved. The root cause was a managed database update that resulted in some database requests failing during a time window of around 5 minutes. May <var data-var='date'>18</var>, <var data-var='time'>10:48</var> UTC Investigating - We are currently investigating failures to login via web browser to the US based Control Room.

Login failures on Control Room US based instance

Mar <var data-var='date'>22</var>, <var data-var='time'>08:06</var> UTC Resolved - We encountered a spike in errors on our rate limiting system during 05:30 EET - 07:30 EET (8:30PM - 10:30PM PDT). The situation is back to normal. The issue may have impacted process related operations like starting, stopping, and listing processes. The system has been patched to prevent similar issues in the future.

Increased error rate on process related features

Nov <var data-var='date'>11</var>, <var data-var='time'>09:31</var> UTC Resolved - The situation is back to normal. Approximately 10% of requests to the task data subsystem between 6:54 UTC and 7:54 UTC were failing with HTTP status code 500. Impacted step runs may have failed depending on which actions the robot has taken. Run failure notifications have been operational during the time, thus surfacing the issue to robot operators. The failed work items can be retried from the control room. We apologize for the inconvenience. Nov <var data-var='date'>11</var>, <var data-var='time'>08:04</var> UTC Investigating - We are currently investigating Internal server error responses from the work item system.

Work Item system failures

Nov <var data-var='date'> 2</var>, <var data-var='time'>13:42</var> UTC Resolved - We encountered a significant spike in errors from AWS services during 14:35 EET - 14:37 EET (5.35 - 5.37 Pacific Time). The situation is back to normal. Based on the inspection of dead-letter-queues, all failures in customer workloads were automatically recovered via retry mechanisms. Nov <var data-var='date'> 2</var>, <var data-var='time'>13:11</var> UTC Investigating - We are currently investigating increased error rates across various AWS services on eu-west-1, where our EU-based Control Room is hosted.

Increased error rates

Oct <var data-var='date'>21</var>, <var data-var='time'>21:02</var> UTC Resolved - We encountered errors with the Slack integration in Control Room this week on Wednesday 19th and Thursday 20th. Customers impacted by failures to deliver Slack notifications have been explicitly contacted along with details of the notifications that were not delivered successfully. The situation has been restored back to normal.

Slack integration failures in Control Room

Oct <var data-var='date'>20</var>, <var data-var='time'>07:30</var> UTC Resolved - We have fixed an issue causing Control Room scheduler to incorrectly start multiple runs for a subset of scheduled runs between Oct 19 16:15 EEST (6.15am PT) and Oct 20 10:20 EEST (1.20am PT). The root cause of the issue was an insufficient timeout configuration, which in some cases caused our resiliency mechanism to trigger a retry before the processing of the original request had been completed. We apologize for any inconvenience.

Multiple runs started by Control Room scheduler

Sep <var data-var='date'>13</var>, <var data-var='time'>10:34</var> UTC Resolved - Bulk of the errors was caused by a change that caused misreporting of 4xx status codes as 5xx. This was visible to clients of the process API, but impacted only requests that failed regardless. Additionally, between 11:49 and 12:05 EEST a subset of requests were failing. We will notify all impacted paid tier customers explicitly during today. Sep <var data-var='date'>13</var>, <var data-var='time'>09:01</var> UTC Investigating - We are investigating increased error rates in process API.

Process API failures

Sep <var data-var='date'> 7</var>, <var data-var='time'>14:00</var> UTC Resolved - We discovered a failure in a subsystem responsible for updating metadata of Step Run Artifacts and Work Item Files. File upload completion detection did not run as expected and associated metadata is out of date, showing files as "not uploaded". The issue impacted file uploads between 17:00 EEST and 18:00 EEST today. The practical impact of this issue is negligible. Files are served regardless of the metadata, and thus there is no issue unless the metadata is explicitly relied on. No data loss has occurred. If you encounter any practical issues, please let us know and we can take action on a case-by-case basis.

Artifact and Work Item File metadata inconsistency

Aug <var data-var='date'>21</var>, <var data-var='time'>10:32</var> UTC Resolved - This incident has been resolved. A backend service was having database connection issues. Minor impact to a small subset of workloads may have occurred. Aug <var data-var='date'>21</var>, <var data-var='time'>09:53</var> UTC Investigating - We are currently observing increased error rates on the platform. All services are still up and running, and we are investigating & monitoring the situation.

Mar <var data-var='date'> 8</var>, <var data-var='time'>17:08</var> UTC Resolved - Situation is back to normal. Our analysis indicates the issues were caused by intermittent failures on our infrastructure provider, but service remained operational at all times due to retries. Mar <var data-var='date'> 8</var>, <var data-var='time'>10:39</var> UTC Investigating - We are currently investigating increased error rates in the platform.

Nov <var data-var='date'>29</var>, <var data-var='time'>14:02</var> UTC Resolved - Issue resolved. We rolled back part of the changes that caused the issues. Nov <var data-var='date'>29</var>, <var data-var='time'>13:58</var> UTC Investigating - We are currently investigating this issue.

Is Robocorp down?

Robocorp status is UP

Lastest published events

Maintenance day

Website and documentation impacted by Cloudflare issues

Issues on US based deployments

Robocorp Services, Components and Regions

Recent Robocorp events

Maintenance day

Website and documentation impacted by Cloudflare issues

Issues on US based deployments

Database engine upgrade on SSO authentication database, US based deployments

Database engine upgrade on SSO authentication database, EU based deployments

Database engine upgrade on eu1 (Ireland) Control Room instance