Mar <var data-var='date'>31</var>, <var data-var='time'>08:53</var> UTC Resolved - Quay.io functionality has been restored and DataRobot environments are fully stabilized.Mar <var data-var='date'>30</var>, <var data-var='time'>20:44</var> UTC Update - We are continuing to work on a fix for this issue.Mar <var data-var='date'>30</var>, <var data-var='time'>20:43</var> UTC Identified - Our engineering team has found the the Quay outage currently happening is causing degraded performance across the DataRobot platform. Engineering is currently monitoring the situation.

Degraded Performance on DataRobot MTS due to Quay outage

Mar <var data-var='date'>17</var>, <var data-var='time'>18:32</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>17</var>, <var data-var='time'>18:31</var> UTC Update - We are continuing to monitor for any further issues.Mar <var data-var='date'>11</var>, <var data-var='time'>20:22</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'>11</var>, <var data-var='time'>20:05</var> UTC Investigating - We are currently investigating this issue.

Intermittent UI disruptions on Managed AI Cloud

Mar <var data-var='date'>13</var>, <var data-var='time'>18:43</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>13</var>, <var data-var='time'>18:31</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'>13</var>, <var data-var='time'>17:49</var> UTC Investigating - We are experiencing performance degradation on Managed AI Cloud.

Performance Degradation on Managed AI Cloud

Mar <var data-var='date'>11</var>, <var data-var='time'>16:41</var> UTC Resolved - The mitigation implemented by Engineering has resolved the Kubernetes network issue, and the incident is now contained.Mar <var data-var='date'>11</var>, <var data-var='time'>15:19</var> UTC Monitoring - The mitigation implemented by Engineering has improved the network issue. The team is continuing to monitor the environment to ensure full recovery.Mar <var data-var='date'>11</var>, <var data-var='time'>15:06</var> UTC Identified - Engineering has identified the root cause of the problem and a mitigation is put in place.Mar <var data-var='date'>11</var>, <var data-var='time'>14:36</var> UTC Investigating - DataRobot is experiencing network issue related to Kubernetes in US Cluster. This will have impact on model deployment and predictions. Engineering is investigating the root cause.

Network issue related to Kubernetes in US cluster

Feb <var data-var='date'>18</var>, <var data-var='time'>21:19</var> UTC Resolved - This incident is now resolved.Feb <var data-var='date'>18</var>, <var data-var='time'>21:04</var> UTC Investigating - Our engineering team has found the the Quay outage currently happening is causing degraded performance across the DataRobot platform.

Degraded Performance on the DataRobot MTS due to Quay outage

Feb <var data-var='date'>17</var>, <var data-var='time'>15:55</var> UTC Resolved - Rollback of the JP cluster to the previous version is complete and the problem has been mitigated.Feb <var data-var='date'>17</var>, <var data-var='time'>15:48</var> UTC Identified - LLM blueprints deployments can not be created in JP MTS environment. Engineering is rolling back JP cluster to previous version to mitigate the issue.

LLM blueprints deployments cannot be created

Feb <var data-var='date'>16</var>, <var data-var='time'>12:49</var> UTC Resolved - New version of Agentic application template is released, the issue is resolvedFeb <var data-var='date'>16</var>, <var data-var='time'>11:35</var> UTC Identified - Agent application template is affected with the recent moderations library upgrade, fix is identified and mitigation is in progress.

Agent Application Template Impacted After Moderations Library Upgrade.

Feb <var data-var='date'>13</var>, <var data-var='time'>21:31</var> UTC Resolved - The incident has now been resolved. All services are now operational.Feb <var data-var='date'>13</var>, <var data-var='time'>20:39</var> UTC Investigating - We are observing issues on DataRobot US MTS environment. Users may experience degraded performance using APIs and data ingest services. The engineering team is currently investigating the root cause.

Degraded Performance on the DataRobot US MTS

Jan <var data-var='date'>22</var>, <var data-var='time'>14:19</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'> 8</var>, <var data-var='time'>18:16</var> UTC Update - We are continuing to monitor for any further issues.Jan <var data-var='date'> 8</var>, <var data-var='time'>18:15</var> UTC Monitoring - Some customers have reported issue connecting to DataRobot. Please do a hard refresh of your browser by clearing the cache and this should fix the problem. As always let us know if it continue to have issue connecting to DataRobot after clearing the cache.

Problem connecting to DataRobot due to client browser caching

Dec <var data-var='date'>29</var>, <var data-var='time'>11:00</var> UTC Completed - The scheduled maintenance has been completed.Dec <var data-var='date'>29</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Dec <var data-var='date'> 9</var>, <var data-var='time'>13:31</var> UTC Scheduled - DataRobot is performing an infra maintenance on app.datarobot.com, app.eu.datarobot.com and app.jp.datarobot.com. During this time, some users might experience intermittent interruptions with DataRobot LLM Gateway OpenAI models. Please reach out to support@datarobot.com if you have any questions.

DataRobot LLM Gateway OpenAI models. Planned Maintenance

Dec <var data-var='date'>17</var>, <var data-var='time'>17:00</var> UTC Resolved - Between 16:50 UTC and 17:09 UTC, one of external providers(DockerHub) had an outage that might have caused temporary delays in starting platform workloads. The issue has been resolved and normal operations have resumed. Engineering is continuing to monitor the system.

Degraded Platform Performance Due to Docker Hub Outage

Dec <var data-var='date'> 5</var>, <var data-var='time'>11:35</var> UTC Resolved - The engineering team has confirmed recovery across all MTS and STS environments. No further updates are expected.Dec <var data-var='date'> 5</var>, <var data-var='time'>11:09</var> UTC Monitoring - Between 09:04 UTC and 09:17 UTC, one of external providers(DockerHub) had an outage that might have caused temporary delays in starting platform workloads. The issue has been resolved and normal operations have resumed. Engineering is continuing to monitor the system.

Nov <var data-var='date'>27</var>, <var data-var='time'>08:26</var> UTC Resolved - This incident has been resolved.Nov <var data-var='date'>26</var>, <var data-var='time'>19:26</var> UTC Update - Our team has deployed a fix for the issue. All the MTS environments are currently operational.Nov <var data-var='date'>26</var>, <var data-var='time'>14:04</var> UTC Update - Engineering team has verified that the fix has been applied to JP MTS production environment and Non GenAI users are now able to access notebooks in JP cluster. We are now applying the hotfix to US and EU clusters. The mitigation for Self Managed and STS deployments is in progress.Nov <var data-var='date'>26</var>, <var data-var='time'>13:30</var> UTC Update - Engineering team has verified that the fix has been applied to JP MTS production environment and Non GenAI users are now able to access notebooks in JP cluster. We are now applying the hotfix to US and EU clusters.Nov <var data-var='date'>26</var>, <var data-var='time'>13:28</var> UTC Update - We are continuing to work on a fix for this issue.Nov <var data-var='date'>26</var>, <var data-var='time'>10:30</var> UTC Identified - Engineering team has identified the issue and hotfix is being applied to all MTS envsNov <var data-var='date'>26</var>, <var data-var='time'>07:17</var> UTC Investigating - Some Users with GenAI licensing are facing issues in accessing Notebooks/Codespaces. Engineering team is checking into it.

Some Users Facing Issues In Accessing Notebooks/Codespaces in US/EU/JP MTS.

Nov <var data-var='date'>20</var>, <var data-var='time'>07:23</var> UTC Resolved - The issue is fixed. Access to the US, EU, and JP clusters is restored.Nov <var data-var='date'>20</var>, <var data-var='time'>06:52</var> UTC Investigating - Engineering is investigating reports of some users not being able to login in US,EU & JP MTS clusters. Please contact support@datarobot.com if you have any questions.

Some Users Are Unable To Login Into US,EU & JP MTS clusters.

Nov <var data-var='date'> 7</var>, <var data-var='time'>12:37</var> UTC Resolved - The Engineering team hasn't observed any new occurrences during monitoring. All services remain fully operational.Nov <var data-var='date'> 6</var>, <var data-var='time'>13:02</var> UTC Monitoring - The issue of Notebooks timing out is not reproducible on US Prod. Engineering is monitoring the service and investigating the root cause.Nov <var data-var='date'> 6</var>, <var data-var='time'>12:38</var> UTC Investigating - Notebooks are timing out occasionally on US Production, engineering team is investigating the issue.

Notebooks are timing out occasionally on US Production.

Nov <var data-var='date'> 7</var>, <var data-var='time'>11:50</var> UTC Resolved - The issue has been resolved.Nov <var data-var='date'> 7</var>, <var data-var='time'>07:14</var> UTC Investigating - The AutoML experiment creation workflow and the visibility of certain tabs in both the Classic and NextGen UIs are currently impacted for the customers with GenAI license. Engineering is actively investigating the issue.

The AutoML experiment creation workflow and visibility of certain tabs impacted in MTS environments

Nov <var data-var='date'> 4</var>, <var data-var='time'>14:32</var> UTC Resolved - The Engineering team has successfully resolved the connection and processing timeouts affecting data operations in MTS Production. Services are now running stable.Nov <var data-var='date'> 4</var>, <var data-var='time'>13:57</var> UTC Investigating - Users in MTS Production are not able to create new Data Connections, the process times out and Data Connection cannot be saved. The issue was reported for Snowflake connection, however other connections may be affected too. The Engineering team is actively investigating the issue.

Data Connectors Hanging on Creation

Oct <var data-var='date'>22</var>, <var data-var='time'>14:40</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>22</var>, <var data-var='time'>14:40</var> UTC Update - The hotfix has been applied and access for Org Admins to Custom Applications has been restored.Oct <var data-var='date'>22</var>, <var data-var='time'>13:16</var> UTC Identified - After the recent permission update, Org Admins temporarily lost access to Custom Applications. A fix is being deployed and should be completed within 6 hours.

Temporary Access Issue for Org Admins to Custom Applications

Oct <var data-var='date'>20</var>, <var data-var='time'>22:11</var> UTC Resolved - This incident has been resolved. AWS is reporting systems as operational. Engineering will continue to monitor the situation.Oct <var data-var='date'>20</var>, <var data-var='time'>13:57</var> UTC Update - We are continuing to monitor for any further issues.Oct <var data-var='date'>20</var>, <var data-var='time'>10:03</var> UTC Monitoring - Overall functionality of DataRobot is recovering. Engineering team is actively monitoring the services to ensure continued stability.Oct <var data-var='date'>20</var>, <var data-var='time'>10:03</var> UTC Update - We are continuing to investigate this issue.Oct <var data-var='date'>20</var>, <var data-var='time'>10:02</var> UTC Update - We are continuing to investigate this issue.Oct <var data-var='date'>20</var>, <var data-var='time'>10:01</var> UTC Update - We are continuing to investigate this issue.Oct <var data-var='date'>20</var>, <var data-var='time'>09:49</var> UTC Investigating - Due to on-going AWS outage in us-east-1 region overall functionality of MTS and STS is affected. Users will experience degraded performance and intermittent connectivity issues. Engineering team is in contact with AWS team to find a resolution.

AWS services outage affects STS and MTS DataRobot environments

Oct <var data-var='date'> 1</var>, <var data-var='time'>18:04</var> UTC Resolved - Engineering team has applied the fix to address UI issue affecting Model Visibility and Project Management. This problem has now been resolved.Oct <var data-var='date'> 1</var>, <var data-var='time'>14:58</var> UTC Update - The Engineering team has released and validated the fix on EU AI Cloud. Deployment to US AI Cloud is in progress.Oct <var data-var='date'> 1</var>, <var data-var='time'>13:14</var> UTC Update - The Engineering team has released and validated the fix on JP AI Cloud. Deployment to EU and US AI Clouds is in progress.Oct <var data-var='date'> 1</var>, <var data-var='time'>09:40</var> UTC Identified - We’ve identified an issue causing limited access to Projects in Classic UI for customers with GenAI Builder seat license assigned. The Engineering team is working on a fix.

UI Issue Affecting Model Visibility and Project Management

Sep <var data-var='date'>25</var>, <var data-var='time'>01:56</var> UTC Resolved - Issue is resolved and DataRobot MTS & STS services are back to normal.Sep <var data-var='date'>25</var>, <var data-var='time'>01:35</var> UTC Monitoring - As of 01:09 UTC (Sept 25)/ 06:09 PM PST, systems have started recovering from the DockerHub outage. DataRobot internal services are back to normal. The engineering team will continue to monitor closely.Sep <var data-var='date'>25</var>, <var data-var='time'>01:19</var> UTC Identified - At 23:09 UTC / 4:09 PM PST, an unexpected outage from DockerHub triggered internal alerts. The engineering team is currently assessing potential impact across environments. We will share further details and mitigation steps as soon as our investigation progresses.

Issue With DockerHub Services Impacting MTS & STS DataRobot Clusters.

Sep <var data-var='date'>10</var>, <var data-var='time'>18:03</var> UTC Resolved - This incident has been resolved.Sep <var data-var='date'> 9</var>, <var data-var='time'>18:14</var> UTC Update - We are continuing to monitor for any further issues.Sep <var data-var='date'> 9</var>, <var data-var='time'>14:02</var> UTC Monitoring - Users can now access US MTS cluster from APAC region and issue recovered. The engineering team will continue monitoring this issue.Sep <var data-var='date'> 9</var>, <var data-var='time'>11:26</var> UTC Update - Users in APAC are experiencing network issues when accessing application endpoints in US MTS. Engineering is investigating potential ISP-related networking issues in APAC region. Affected customers are encouraged to try using a different network and/or browsers to mitigate the issue.Sep <var data-var='date'> 9</var>, <var data-var='time'>05:39</var> UTC Investigating - We've identified that some users in Japan are experiencing an intermittent login issue when trying to access the US MTS. The engineering team is currently investigating.

Users in APAC region are experiencing network issues when accessing application endpoints in US MTS

Sep <var data-var='date'> 2</var>, <var data-var='time'>18:07</var> UTC Resolved - Engineering has applied the fix to mitigate the issue impacting serverless deployments when a new deployment is created or an existing deployment is modified. The problem has now been contained.Sep <var data-var='date'> 2</var>, <var data-var='time'>15:37</var> UTC Investigating - We've identified an issue impacting serverless deployments when a new deployment is created or an existing deployment is modified, in Japan cluster. Prediction against existing deployments deployed to serveless prediction server that are not modified are not impacted. Engineering currently working on a fix to address the issue.

Serverless Prediction Servers are impacted when an a new deployment is created or an existing deployment is modified.

Jul <var data-var='date'>29</var>, <var data-var='time'>22:09</var> UTC Resolved - This incident has been resolved.Jul <var data-var='date'>29</var>, <var data-var='time'>14:32</var> UTC Identified - Engineering has identified the root cause and prepared a fix. The team has now started deploying the fix to MTS production.Jul <var data-var='date'>29</var>, <var data-var='time'>11:51</var> UTC Investigating - Some of the customers are not receiving scheduled deployment reports. The data can still be observed in the application UI. The engineering is actively working on restoring the functionality. Please check the below-mentioned link for docs for generating report: https://docs.datarobot.com/en/docs/mlops/governance/deploy-reports.html#generate-a-deployment-report

Some of the customers are not receiving scheduled deployment reports.

Jul <var data-var='date'> 7</var>, <var data-var='time'>16:20</var> UTC Resolved - The fix has resolved the interface failure issue. DataRobot App in US Prod is fully functioning as normal.Jul <var data-var='date'> 7</var>, <var data-var='time'>16:12</var> UTC Monitoring - The fix has been applied successfully and the interface has been restored. Engineering is currently monitoring the system.Jul <var data-var='date'> 7</var>, <var data-var='time'>15:42</var> UTC Identified - Starting 11:05 EST today, Interface is down for all customers in the US Prod due to internal error. Engineering has identified the problem and is in the process of applying the fix.

Is DataRobot down?

DataRobot status is UP

Lastest published events

Degraded Performance on DataRobot MTS due to Quay outage

Intermittent UI disruptions on Managed AI Cloud

Performance Degradation on Managed AI Cloud

DataRobot Services, Components and Regions

Recent DataRobot events

Degraded Performance on DataRobot MTS due to Quay outage

Intermittent UI disruptions on Managed AI Cloud

Performance Degradation on Managed AI Cloud

Network issue related to Kubernetes in US cluster

Degraded Performance on the DataRobot MTS due to Quay outage

LLM blueprints deployments cannot be created

Get your FREE account now,
5 monitors included!

Is DataRobot down?

DataRobot status is UP

Lastest published events

Degraded Performance on DataRobot MTS due to Quay outage

Intermittent UI disruptions on Managed AI Cloud

Performance Degradation on Managed AI Cloud

DataRobot Services, Components and Regions

Recent DataRobot events

Degraded Performance on DataRobot MTS due to Quay outage

Intermittent UI disruptions on Managed AI Cloud

Performance Degradation on Managed AI Cloud

Network issue related to Kubernetes in US cluster

Degraded Performance on the DataRobot MTS due to Quay outage

LLM blueprints deployments cannot be created

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!