Resolved — The issue has been resolved.
Identified — Email delivery has resumed and should be working again. We are investigating whether we can add further safeguards to prevent re-occurrence.
Investigating — We are currently investigating an issue with our email delivery service. At the moment, our platform is unable to send emails to customers.
This affects all system-generated emails, including account verification emails and invitations to collaborate on projects. Until the issue is resolved, these emails will not be delivered.
We will post an update as soon as we have more information. We apologize for the inconvenience.

Issues sending emails to customers

Completed — All systems are nominal
Verifying — Maintenance has been completed successfully.
Read-only mode has been disabled. All operations have returned to normal.
We are monitoring the system. In case of problems, please contact support.
In progress — Maintenance is still in progress and is taking longer than initially planned.
The platform remains in read-only mode. Calls to graphql.datocms.com continue to work correctly.
We apologize for the inconvenience and will post another update as soon as the maintenance is complete.
In progress — Maintenance is still in progress.
The platform remains in read-only mode. All write operations continue to return a 503 error with code &quot;PLATFORM_SCHEDULED_MAINTENANCE&quot;, while calls to graphql.datocms.com keep working correctly.
We will post another update as soon as the maintenance is complete.
In progress — Maintenance has begun.
The platform is now in read-only mode. All write operations will return a 503 error with code &quot;PLATFORM_SCHEDULED_MAINTENANCE&quot;
Calls to graphql.datocms.com continue to work correctly without interruptions.
There may be brief periods of time when the platform will stop responding to all requests, even read-only ones. In this scenario, calls to graphql.datocms.com already cached in our CDN will continue to work anyway.
We will continue to update the status page as soon as we have updates.
Scheduled — We are going to perform a database upgrade.
📅 Timeline:
<ul>
<li>Full maintenance window begins: September 19th, 2026 05:30 UTC</li>
<li>Full maintenance window expected duration: 3 hours</li>
</ul>
⚠️ Service Restrictions (Starting 05:30 UTC):
<ul>
<li>Consider that for the full 3h window the system will be read-only</li>
<li>All write operations will return an error: content updates, schema changes, upload management, and every other write operation on environments</li>
<li>Read operations on the Content Delivery API and the Content Management API will continue to work</li>
<li>There is going to be a limited-time window in which the system might be completely down. In that scenario, calls to graphql.datocms.com already cached in our CDN will continue to work anyway.</li>
</ul>
We will provide more status updates during the maintenance window. 
Thank you for your understanding.

🛠️ Scheduled database maintenance [ ⚠️ READ-ONLY MODE ]

Resolved — This is now resolved. A date handling bug was hiding the Third-Party Components section whenever a provider had an ongoing incident, and several of the provider feeds we relied on had been discontinued upstream. Both are fixed, and the section now reads each provider&#39;s status API directly.
Investigating — Separate from the Imgix outage, our status page (this current page) is not correctly showing our upstream provider statuses. We are investigating.
Resolved — Earlier today, from approximately 15:00 to 19:00 UTC, our upstream image provider (Imgix) had a partial outage that resulted in occasional rendering errors for some customers in some situations. The issue should now be resolved (see details at <a href="https://status.imgix.com/incidents/ch5p1nlqnckd">https://status.imgix.com/incidents/ch5p1nlqnckd</a> and <a href="https://status.imgix.com/incidents/7n22y1mt7jw1">https://status.imgix.com/incidents/7n22y1mt7jw1</a>

Imgix partial outage and rendering errors

Resolved — The issue has been resolved.
Monitoring — Starting from August 9th at 16:00 UTC, our platform was unable to send emails to customers. This affected all system-generated emails, including account verification and billing notifications.
Email delivery was restored today, August 10th, at around 14:30 UTC, and emails are now being sent normally. No emails have been lost: messages generated during this period have been queued and are being delivered.
We are monitoring the situation and we apologize for the inconvenience.

Resolved — A node in our production Kubernetes cluster became unresponsive, causing a brief disruption to some services while workloads were automatically rescheduled to healthy nodes.
The duration of the disruption was about 8 minutes.

Increase in error rate and response time

Resolved — Closing this.
Monitoring — Starting from 8:15 AM UTC we received alerts about customers not being able to upload new assets in the media area of their project.
This issue only affected uploads made from the administrative area. Uploads performed via the JavaScript client were not impacted.
The issue was traced to a recent deploy. We identified and reverted the problematic changes at 9:20 AM UTC. The issue has been resolved and asset uploads are functioning normally.
We apologize for the inconvenience.

Asset uploads temporarily unavailable

Resolved — Closing the pinned alert on status
Monitoring — A bug prevented daily bandwidth and API call metrics from being saved between January 7 and 12. 
The data was not lost — it has been aggregated into January 13&#39;s totals.
Daily metrics will be recorded normally starting January 14.
We&#39;ve added regression tests to prevent this from recurring. We apologize for the inconvenience.

Daily usage metrics not recorded for January 7–12

Resolved — All issues affecting uploads to DatoCMS using Cloudflare R2 in the EU region have now been fully resolved.
Uploads are operating normally again across the default EU R2 storage as well as custom storage configurations. We are no longer seeing errors related to this incident.
Monitoring — The issue affecting uploads to DatoCMS using Cloudflare R2 in the EU region appears to be resolved at this time.
Uploads are currently working as expected, including on the default Cloudflare R2 (EU) storage. We are continuing to monitor the situation closely to ensure stability.
Investigating — We’re currently investigating an issue affecting the ability to upload new assets to DatoCMS.
The issue impacts Cloudflare R2 buckets in the EU jurisdiction, which is the default storage configuration.
Uploads to custom storage solutions are working as expected, indicating the problem is specific to Cloudflare R2 in the EU region.
Our team is actively investigating the root cause and working with the relevant providers to restore full functionality as quickly as possible. We will share further updates as soon as we have more information.

Issues Uploading New Assets

Resolved — On December 2 at 13:13 CET, our API was unavailable for 9 minutes during a deployment, causing errors when accessing the CMS interface.
What happened
During a routine database migration deployment, code that depended on a not-yet-created database table was accidentally included. This caused the API to fail until we identified and rolled back the changes.
Prevention
We&#39;re implementing stricter code review checks to ensure migration-only deployments don&#39;t include dependent code, and adding automated tests to catch this type of issue before deployment.
We apologise for the disruption and appreciate your patience.

Incident Report - API Outage on December 2, 2025

Resolved — Post-mortem by Cloudflare: <a href="https://blog.cloudflare.com/18-november-2025-outage/">https://blog.cloudflare.com/18-november-2025-outage/</a>
Resolved — The outage should be resolved.
Monitoring — As of 14:42 UTC (15:42 CEST/06:42 PST), Cloudflare believes the issue to be resolved. All DatoCMS services should be back up.
We will continue to monitor, just in case.
If you need help, please email <a href="mailto:support@datocms.com">support@datocms.com</a>.
Monitoring — Services are gradually coming back up, but we don&#39;t know yet if this is a stable fix or if there will be further downtime.
APIs, datocms-assets.com, dashboard, project admin area, and forum are back up for the time being.
EDIT: As of 14:42 UTC, Cloudflare believes services are restored: <a href="https://www.cloudflarestatus.com/incidents/8gmgl950y3h7">https://www.cloudflarestatus.com/incidents/8gmgl950y3h7</a>. We will keep monitoring to be sure.
Please keep an eye here for further updates, join our Slack (<a href="https://join.slack.com/t/datocms/shared_invite/zt-3j142172p-nou_g2g5vUAPL6cjOOz5zQ">https://join.slack.com/t/datocms/shared_invite/zt-3j142172p-nou_g2g5vUAPL6cjOOz5zQ</a>), or reach out to us at <a href="mailto:support@datocms.com">support@datocms.com</a> if you need help.
Investigating — The Cloudflare outage has escalated, and is now causing service disruptions across DatoCMS. We are experiencing outages with:
<ul>
<li>DatoCMS APIs (CDA and CMA)</li>
<li>datocms-assets.com</li>
<li>The dashboard and admin area</li>
<li>The community forum</li>
<li>The status page uptime indicators are stuck at &quot;loading&quot;</li>
</ul>
Our Slack remains available: <a href="https://join.slack.com/t/datocms/shared_invite/zt-3j142172p-nou_g2g5vUAPL6cjOOz5zQ">https://join.slack.com/t/datocms/shared_invite/zt-3j142172p-nou_g2g5vUAPL6cjOOz5zQ</a>
We will update as soon as we can.
Investigating — We are currently experiencing degraded performance caused by an ongoing Cloudflare incident. Due to issues within Cloudflare’s infrastructure, a percentage of API requests are not being properly routed to our servers. This may result in intermittent errors or failed requests for some customers.
Our systems remain fully operational, and we are actively monitoring the situation while Cloudflare works to resolve the upstream problem. We will provide updates as soon as more information becomes available.
More info on: <a href="https://www.cloudflarestatus.com/incidents/8gmgl950y3h7">https://www.cloudflarestatus.com/incidents/8gmgl950y3h7</a>

Major DatoCMS Outages Caused by Upstream Cloudflare Incident

Completed — Monitoring is completed
Verifying — Maintenance completed. We are monitoring the system.
Scheduled — We are going to run a maintenance on our billing infrastructure starting from 2025-10-14T05:00 UTC. The expected time to finish is 2 hours.
During this maintenance window you could not be able to succesfully complete some operations such as:
<ul>
<li>make changes to your plan (buying a plan, upgrading / downgrading a plan, etc.)</li>
<li>update your billing profile (address, credit cards, etc.)</li>
<li>tranfer a project</li>
<li>destroy an account or an organization</li>
<li>destroy a project</li>
<li>pay overdue amounts</li>
</ul>
Automatic plan adjustments that we apply to your plan, such as adding an extra collaborator, will be executed after the maintenance.

Maintenance on our billing system

Completed — All systems are nominal
Verifying — Maintenance has been completed successfully.
Read-only mode has been disabled. All operations have returned to normal.
We are monitoring the system. In case of problems, please contact support.
In progress — Maintenance has begun.
The platform is now in read-only mode. All write operations will return a 503 error with code &quot;PLATFORM_SCHEDULED_MAINTENANCE&quot;
Calls to graphql.datocms.com continue to work correctly without interruptions.
There may be brief periods of time when the platform will stop responding to all requests, even read-only ones. In this scenario, calls to graphql.datocms.com already cached in our CDN will continue to work anyway.
We will continue to update the status page as soon as we have updates.
Scheduled — Maintenance is about to start
Scheduled — We are going to perform an important system upgrade.
📅 Timeline:
<ul>
<li>Full maintenance window begins: August 30th, 2025 05:50 UTC</li>
<li>Full maintenance window expected duration: 2 hours</li>
</ul>
⚠️ Service Restrictions (Starting 05:50 UTC):
<ul>
<li>Consider that for the full 2h window the system will be read-only</li>
<li>There is going to be a limited-time window in which the system might be completely down.</li>
</ul>
We will provide more status updates during the maintenance window. 
Thank you for your understanding.

🛠️ Scheduled maintenance [ ⚠️ DOWNTIME REQUIRED ]

Resolved — Monitoring completed
Monitoring — Between 07:18 UTC and 07:28 UTC a faulty deploy caused degraded performance on our infrastructure. Customers might have noticed increased error rates when trying to access the admin interface, CMA (REST API) and CDA (graphql API).
We reverted the deploy and we are monitoring the situation

Increased error rates

Resolved — On August 4th, 2025, starting from 14:00 UTC, some customers reported issues building their Gatsby projects that use our gatsby-source-datocms plugin due to records being missing from their GraphQL responses, despite being present in the CMS admin area.
The issue was related to a recent change in our code that caused pagination issues with the records fetched by our gatsby-source-datocms plugin. This prevented some records from being visible in the Gatsby GraphQL responses.
A rollback of that change was deployed on August 5th at 7:15 UTC.
A regression test has been added to ensure that this particular issue with pagination won&#39;t happen again.

[Retrospective] Gatsby project build failures caused by missing records in GraphQL responses

Resolved — The developers traced the issue to an auto-scaling issue in a part of our infrastructure, where a cluster wasn&#39;t properly adding more resources as quickly as needed in order to keep up performance under load. We&#39;ve since tweaked the scaling configuration and it should not happen again. Our apologies for the slowdown!
Resolved — Several customers reported intermittent API performance degradation starting from about 1530 UTC (1730 CEST, 8:30am PST) lasting about an hour. Performance appears to be restored now, but we will continue to monitor the situation and investigate the underlying cause. Please email <a href="mailto:support@datocms.com">support@datocms.com</a> if you are still experiencing any performance issues.

2025-07-15: Performance degradation

Resolved — The incidents should be resolved.
Monitoring — On 2025-06-12, both Cloudflare and Google Cloud started having broad outages:
 <a href="https://www.cloudflarestatus.com/incidents/25r9t0vz99rp">https://www.cloudflarestatus.com/incidents/25r9t0vz99rp</a>
 <a href="https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW">https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW</a>
This is in turn impacting the availability and performance of some of our services, both in the CMS and the API. It doesn&#39;t seem like a full outage yet, but general slowdowns so far.
These are in turn also affecting our image CDN: <a href="https://status.imgix.com/incidents/ww7rgnbpkqg7">https://status.imgix.com/incidents/ww7rgnbpkqg7</a>
And our videos CDN: <a href="https://status.mux.com/incidents/8l2vdkxgs0pp">https://status.mux.com/incidents/8l2vdkxgs0pp</a>
This is an evolving situation and we&#39;ll post updates as we can.
It is extremely uncommon for major different internet providers to have simultaneous broad outages like this... we are not sure what is going on either, but we will try to post any information we get.

2025-06-12 Major Cloudflare and Google Cloud Outages affecting service performance

Completed — The scheduled maintenance has been completed and all services are fully operational.
Verifying — Maintenance has been completed successfully.
Read-only mode has been disabled. All operations have returned to normal.
We are monitoring the system. In case of problems, please contact support.
In progress — Maintenance has begun.
The platform is now in read-only mode. All write operations will return a 503 error with code &quot;PLATFORM_SCHEDULED_MAINTENANCE&quot;
Calls to graphql.datocms.com continue to work correctly without interruptions.
There may be brief periods of time when the platform will stop responding to all requests, even read-only ones. In this scenario, calls to graphql.datocms.com already cached in our CDN will continue to work anyway.
We will continue to update the status page as soon as we have updates.
Scheduled — We are going to perform an important system upgrade.
📅 Timeline:
<ul>
<li>Full maintenance window begins: June 7, 2025 04:00 UTC</li>
<li>Full maintenance window expected duration: 4 hours</li>
</ul>
⚠️ Service Restrictions (Starting 04:00 UTC):
<ul>
<li>Consider that for the full 4h window the system will be read-only</li>
<li>There is going to be a limited-time window in which the system might be completely down.</li>
</ul>
We will provide more status updates during the maintenance window. 
Thank you for your understanding.

Resolved — Postmortem: we made some changes in our Cloudflare configuration to enable a higher cache hit ratio. We relied on the fact that Cloudflare explicitly states that they support URLs up to 16KB long: <a href="https://developers.cloudflare.com/workers/platform/limits/#request-limits">https://developers.cloudflare.com/workers/platform/limits/#request-limits</a> Turns out it&#39;s not true: we asked for more info from Cloudflare; in the meantime, we reverted our optimization.
Monitoring — Our services experienced increased response times for 12 minutes between 4:25 CEST and 4:37 CEST today. The issue was promptly identified and resolved by our engineering team.
Please note that the status page graphs incorrectly show a +50-minute degradation period due to a misconfiguration in our reporting system. This does not reflect the actual duration of the incident. The reporting system has been corrected. All services are now operating normally with regular response times.

Response Time Degradation

Resolved — The issue has been resolved.
Investigating — PDFs served from datocms-assets.com are currently showing up as image thumbnails rather than PDF files for Chromium-based browsers (Chrome and Edge). Firefox and Safari are not affected.
A current workaround is to append <code>?skip-default-optimizations</code> to the end of the URL, like:
<a href="https://www.datocms-assets.com/123456/123456789-your-file.pdf?skip-default-optimizations">https://www.datocms-assets.com/123456/123456789-your-file.pdf?skip-default-optimizations</a>
That should allow the PDF to load normally.
We are working on a permanent resolution and will update here as soon as we can.

PDFs rendering as thumbnail images on Chrome & Edge

Resolved — We&#39;ve hotfixed the issue again for now. Sincerely apologize for the regression. We&#39;ll be more careful next time.
Identified — The issue is intermittent, and the login is working again for the time being. We are working with our third-party partner to resolve the underlying issue.
Identified — The issue has resurfaced and the login page is having issues again. We are working on it. Sorry!
Our APIs are not affected.
Resolved — This issue should be resolved.
Monitoring — Login should be working now after the latest hotfix. Please close your browser and try again.
We&#39;ve temporarily disabled a problematic script that was causing this, and will roll out a full fix as soon as we can. We don&#39;t expect any additional downtime from the fix.
Investigating — We’ve received and confirmed some reports about the login page (<a href="https://dashboard.datocms.com/login">https://dashboard.datocms.com/login</a>) not working, resulting in a hanged browser window.
We are trying to find a fix. The problem seems to be intermittent and related to how we handle some third-party network issues.
<ul>
<li>The best temporary workaround seems to be to block posthog.com in your adblocker (if you have one) or browser inspector or system hosts file.
</li>
<li>If you can’t do that, sometimes forcibly reloading the page (you might have to close the whole browser and try again) several times in a row will work. The issue seems to be intermittent.
</li>
</ul>
Our APIs do not seem to be affected. Your websites should continue to work.
Devs have been notified and we’ll update as soon as we can.

Login page not working, causing browser stalling

Completed — The monitoring phase has ended
Verifying — Maintenance is completed. The environments are now in write mode.
We are monitoring the situation.
In progress — The required downtime is over. Now the environments will be in read-only mode for the rest of the maintenance window.
In progress — Maintenance is starting now. 
In progress — Pre maintenance restrictions now apply
The following operations will return 403 errors throughout the maintenance period:
<ul>
<li>Creating sandbox environments</li>
<li>Destroying sandbox environments</li>
<li>Project duplication</li>
<li>Project deletion</li>
</ul>
Scheduled — 🔄 Based on your feedback, we have revised our maintenance approach to minimize service disruption
Key Changes:
<ul>
<li>Reduced downtime: system will only be offline for the first 10 minutes (starting at 09:00 UTC)</li>
<li>After the initial 10-minute downtime, services will be available in read-only mode for the remaining 50 minutes</li>
<li>Total duration increased from 45 to 60 minutes</li>
</ul>
Please refer to the maintenance schedule and impact sections above for complete details. Thank you for your understanding.
Scheduled — We will be performing critical database maintenance that will affect our services.
📅 Timeline:
<ul>
<li>Pre-maintenance restrictions begin: January 26, 2025 08:45 UTC</li>
<li>Full maintenance window begins: January 26, 2025 09:00 UTC</li>
<li>Full maintenance window expected duration: 60 minutes</li>
</ul>
⚠️ Service Restrictions (Starting 08:45 UTC):
The following operations will return 403 errors throughout the maintenance period:
<ul>
<li>Creating sandbox environments</li>
<li>Destroying sandbox environments</li>
<li>Project duplication</li>
<li>Project deletion</li>
</ul>
⚠️ Full Maintenance Impact (Starting 09:00 UTC):
Phase 1 - System Downtime (First 10 minutes):
<ul>
<li>Content Management API: Complete outage</li>
<li>Content Delivery API: Complete outage (cached CDN requests will still work)</li>
<li>Admin Interface: Complete outage</li>
<li>Assets URLs: Fully functional</li>
</ul>
Phase 2 - Restricted Access (Remaining duration):
The system will return online with all environments in maintenance mode:
<ul>
<li>Read operations: Fully functional</li>
<li>Write operations: Temporarily disabled, returning 403 errors for operations including content updates, schema changes, upload management, all other write operations on environments.</li>
</ul>
We will provide more status updates during the maintenance window. 
Thank you for your understanding.

🛠️ Scheduled database maintenance [ ⚠️ DOWNTIME REQUIRED ] - EDITED

Completed — All good 🎉
Verifying — Maintenance completed. We are monitoring
In progress — Maintenance in progress
Scheduled — Maintenance will start in about one hour, like planned
Scheduled — We&#39;re going to upgrade our databases with a follower changeover strategy.
During this period of time the Content Management API, Content Delivery API and projects administrative interface will be completely unavailable.
Cached GraphQL requests in our CDN will continue to work.
The actual maintenance impact should be contained to 15 minutes starting from 2024-12-15T09:00:00Z (UTC).

Scheduled database maintenance [DOWNTIME REQUIRED]

Resolved — This issue can be considered resolved
Monitoring — Between 2024-11-07 12:57 UTC and 13:03 UTC customers have noticed an increase of error rates when accessing their projects admin interface, CMA endpoint and CDA endpoint.
This was caused by a temporary increase of activity on one of our databases.
The situation went back to normal at about 13:03 UTC and we are monitoring the situation.
Cached graphql queries in our edge network were not affected by this.
We are sorry for the inconvenience.

Resolved — Our upstream partner published their post-mortem (Root Cause Analysis (RCA)) about this downtime. You can read the full post here: <a href="https://status.heroku.com/incidents/2684">https://status.heroku.com/incidents/2684</a>
Summary:
On October 8, 2024 from 16:27 Coordinated Universal Time (UTC) to 22:10 UTC, and again on October 9, 2024 from 13:06 UTC to 16:36 UTC, a Distributed Denial of Service (DDoS) caused routing failures in a single partition of Heroku’s Common Runtime infrastructure in the EU Region. This resulted in increased error rates and latencies for some clients connecting to some customer applications hosted in that partition. The Salesforce Technology team worked with our upstream infrastructure provider to mitigate the immediate impact of this event, and to put additional network-level protections in place to improve resilience.
Resolved — Heroku resolved the issue: <a href="https://status.heroku.com/incidents/2685">https://status.heroku.com/incidents/2685</a>
We are still waiting for their post-mortem message about these two incidents.
Identified — The upstream provider is working on a fix
Investigating — The upstream provider is currently investigating the issue. This is their new status page issue: <a href="https://status.heroku.com/incidents/2685">https://status.heroku.com/incidents/2685</a>
Investigating — It looks like Heroku is having the same problem again. We have already contacted them.
Resolved — We are waiting for Heroku&#39;s post mortem in order to collect more information about the downtime
Resolved — The upstream issue has been resolved and services are fully restored. Their last update:
<blockquote>
Starting at 4:27 PM UTC on October 8th , 2024, customers experienced increased error rates and latencies on customer applications hosted on common runtime in EU Region. Heroku engineers investigated and mitigated the impact at 9:52 PM UTC on October 8th , 2024. All application have fully recovered by 10:10 PM UTC.
</blockquote>
Monitoring — Services (CMA, UI, CDA) appear to be coming back up and are accessible at normal performance again.
However, we still have not yet received an official confirmation of resolution from the upstream provider. We will continue to monitor until an official resolution is announced.
Identified — The upstream provider is still investigating the issue. We will continue to post updates here, or you can see their status directly at <a href="https://status.heroku.com/incidents/2684">https://status.heroku.com/incidents/2684</a>
Informational note: Already-cached Content Delivery API (GraphQL) requests do not seem to be affected. Uncached CDA queries may or may not be impacted, depending on the specific query and region.
We are still awaiting a resolution and will continue to monitor the situation.
Identified — The upstream provider identified an issue on their infrastructure: <a href="https://status.heroku.com/incidents/2684">https://status.heroku.com/incidents/2684</a>
Investigating — We are still investigating the issue, and have contacted an upstream provider. We will update again once we have more information.
Investigating — We are currently investigating the issue

High response times on CMA. Admin interface not loading. CDA partially impacted.

Completed — Everything is OK, we can mark this maintenance as resolved.
Verifying — Maintenance completed. We are monitoring.
In progress — Maintenance is in progress!
Scheduled — The scheduled maintenance will start in about 2 hours
Scheduled — We&#39;re going to upgrade our databases with a follower changeover strategy.
During this period of time the Content Management API, Content Delivery API and projects administrative interface will be completely unavailable.
Cached GraphQL requests in our CDN will continue to work.
The actual maintenance impact should be contained to 15 minutes starting from 2024-10-06T08:00:00Z (UTC).

Resolved — This issue can be closed now.
Monitoring — We noticed that since 10:30 UTC webhooks have been delivered with some excessive delay. The queue of webhooks went back to normal at 13:30 UTC. We have identified a possible cause and we are working on a permanent solution. At the same time we are monitoring the situation.

[Retrospective] Webhooks delay

Completed — Monitoring is concluded.
Verifying — Maintenance completed. Now we are monitoring.
In progress — Maintenance in progress
Scheduled — Maintenance will start at planned time
Scheduled — We&#39;re going to upgrade our databases with a follower changeover strategy.
During this period of time the Content Management API, Content Delivery API and projects administrative interface will be completely unavailable.
Cached GraphQL requests in our CDN will continue to work.
The actual maintenance impact should be contained to 15 minutes starting from 2024-07-14T08:00:00Z (UTC).

Resolved — From 4 AM UTC to 11 AM UTC, a subset of assets stored on DatoCMS experienced unresponsiveness, returning 404 errors. The impact was limited, with less than 1% of total assets being affected. We have identified and addressed the root cause, and all affected assets are now functioning correctly. 
We will continue to monitor the situation closely to ensure the sustained stability of DatoCMS. If you have any lingering concerns or notice any issues, please reach out to our support team for further assistance.
Identified — We are noticing rendering errors for some assets hosted on <code>www.datocms-assets.com</code>
We have already identified the issue and we are fixing it

Rendering errors on some assets

Resolved — The upstream DNS provider marks the issue as full resolved.
Monitoring — We got reports of request erroring for CMA and CDA endpoints. The administration interface was affected too, since it uses the CMA endpoint underneath.
We have identified an issue with an upstream DNS provider. They are already aware of the issue.
The problem started at about 2023-11-08 15:45:07 UTC and lasted for about 7 minutes.
We are monitoring the situation

Errors on CMA and CDA requests

Resolved — This incident was caused by a customer with a large user base using a private plugin that went rogue. 
The plugin started making a lot of API calls and since the user base of this customer is very large the entire platform was hit by a DDoS. This caused generic slow downs as the requests were authenticated as users and were hitting directly our web servers.
To avoid this from happening again we have added rate-limitations to JWT tokens, that are used by plugins acting as users, something that we didn&#39;t anticipate before.
Resolved — We can close the issue now
Monitoring — We noticed another batch of unusual traffic at reported time.
We will deploy some countermisures soon.
Monitoring — We experienced about 10 minutes of high response times on our CMA and CDA endpoints. This means that requests sent to those endpoints (site-api.datocms.com, graphql.datocms.com) started to have high latencies or timeout errors (503). 
The admin interface was affected too, since it uses the CMA endpoint itself.
The cause was as unusual amount of traffic that we are going to investigate further.
We are monitoring the situation right now.

High response times on CMA and CDA - Admin interface not loading

Resolved — A code change in the CMS conflicted with some plugins which made the user-interface unusable.
This didn&#39;t affect the APIs and only records that used certain plugins.
As soon as we got notified we issued a fix in the CMS. Unfortunately opening to third-party plugins opens to the possibility of complex scenarios that we haven&#39;t thought about. We have learnt something new from this incident and we&#39;ll try to prevent these issues from happening again in the future.
Resolved — We can close this issue now.
Monitoring — This should be fixed right now. We are monitoring the situation
Investigating — We have received reports of issues while opening records with fields having plugins.
Our REST API (site-api.datocms.com) and CDA (graphql.datocms.com) is not affected.

High error rate on admin interface

Resolved — We have further analyzed what happened in this incident and here&#39;s a longer explanation.
We use Cloudflare workers at the edge of our platform to cache GraphQL requests and block traffic that goes over hard limits, for example for people with cancelled subscriptions. To block these projects we save the information in the Cloudflare KV.
When Cloudflare KV was down our workers were breaking causing the cache to be bypassed.
So all the GraphQL requests were going straight to our servers causing an unusual load that slowed down everything very significantly. Only a minority of requests timed out, but still the user experience was significantly affected and everything seemed down. Furthermore, because everything hit our servers the rate-limits for GraphQL requests were hit much more frequently.
To avoid this from happening in the future we are going to improve our workers to avoid them for breaking in case of the KV service being down. If this has to happen again at least the caching will keep working. We didn&#39;t anticipate that the workers and KV storage might break independently, but we&#39;ll improve this part of the infrastructure soon.
Resolved — This incident is now resolved. The impact start time was 2023-10-30, 20:03 UTC, and the end time was 2023-10-30, 20:35 UTC.
The incident caused response times to be longer than usual on CDA API. We will investigate the issue, and we&#39;ll get back here with more details.

Cloudflare: Cloudflare Pages, Workers KV and Cloudflare Access Availability Issues

Resolved — This incident is now resolved. The impact start time was 2023-09-27 18:30 UTC, and the end time was 2023-09-27 20:10 UTC.
Identified — We confirm that the issue is related to the following Heroku incident: <a href="https://status.heroku.com/incidents/2590">https://status.heroku.com/incidents/2590</a>
We&#39;re trying to investigate if we can circumvent the underlying platform issue.
Identified — The issue seems to be related to Platform Issues on the EU region of the Heroku platform. We are following the issue and will share an update as we have more information.
You can follow the underlying problem on the status page of Heroku as well: <a href="https://status.heroku.com/">https://status.heroku.com/</a>
Investigating — Users are experiencing timeouts when attempting to access the Real-Time API. This incident report is being generated to document the issue and communicate that we are actively investigating the root cause.

Unresponsive Real-Time APIs

Resolved — On September 8th Rollbar announced a security incident.
Rollbar is our error monitoring tool, where we send stacktraces of errors collected on frontend and backend systems.
All the details of the incident here: <a href="https://rollbar.com/compliance/nob_sep7th/">https://rollbar.com/compliance/nob_sep7th/</a>
In brief, our Rollbar access tokens might have been accessed by a third party, but Rollbar has no evidence of this happening. This means that our traces could have been read by someone else. In our implementation, before sending the traces to Rollbar we strip any personal information that might be present or any access token to DatoCMS projects.
Apart from the posted data in the traces nothing else could have been exposed to third parties.
In any case, on Sep 12th we have rotated all our tokens and deleted all the stack traces that were present in Rollbar. To this day we don&#39;t have any evidence that any of our data was accessed by a malicious third party.

Rollbar security notice

Resolved — A fix has been implemented by Fastly
Identified — We have received reports from our customers of failing builds on Vercel with error 502 I/O Error while performing requests to graphql.datocms.com
It looks like that our CDN provider (Fastly) has an open issue related to the San Jose California datacenter: <a href="https://www.fastlystatus.com/incident/375837">https://www.fastlystatus.com/incident/375837</a> 
Since Vercel is building your projects from San Francisco, it&#39;s very likely that they will end to the San Jose California (SJC) Fastly datacenter.
This is not related to Vercel. You might get these errors if you build your project in California.

Builds failing on Vercel (or near San Jose California USA)

Completed — Everything is OK at the moment.
Verifying — Maintenance completed. We are monitoring.
In progress — Maintenance started.
During this period of time the Content Management API, Content Delivery API and projects administrative interface will be completely unavailable.
Cached GraphQL requests in our CDN will continue to work.
Scheduled — Maintenance starting in 1 hour
Scheduled — We&#39;re going to upgrade our databases with a follower changeover strategy.
During this period of time the Content Management API, Content Delivery API and projects administrative interface will be completely unavailable.
Cached GraphQL requests in our CDN will continue to work.
The actual maintenance impact should be contained to 60 minutes starting from 2023-04-29T13:00:00Z (UTC).

Resolved — We can close the issue now.
Monitoring — We have fixed the issue and we are monitoring the system. The issues a global purge in the meantime.
Identified — We identified the issue that caused the GraphQL request to not being purged correctly.
Investigating — We started to investigate the problem.
Identified — We received some reports about GraphQL requests returning stale content.

Delay in CDN purging: serving stale GraphQL data

Completed — Maintenance completed successfully: all systems are back to normal.
Verifying — Maintenance completed. Monitoring
In progress — We are ready to perform the maintenance.
Scheduled — We&#39;re going to upgrade our systems during a short maintenance window.
The Content Management API, Content Delivery API and project administrative interface will be unavailable during this time. The actual maintenance impact should be contained to 30 minutes.

Scheduled maintenance

Resolved — The incident has been closed.
Monitoring — Servers load is within acceptable range now. We are still monitoring the situation.
Investigating — We are experiencing significant difficulties in handling the volume of requests being sent to our API. 
We are actively monitoring the situation and working to implement the most effective solutions. We apologize for any inconvenience this may be causing our users and appreciate your patience as we work to resolve this issue.
If you have any questions or concerns, please do not hesitate to reach out to our support team for assistance. We will continue to provide updates as we make progress in addressing this issue.

Issues with CDA/CMA Request Handling

Completed — Maintenance completed successfully. Billing related changes are back to normal.
Verifying — Maintenance has finished. Chargebee is monitoring, once they confirm everything has succeeded we&#39;ll provide an update ourselves.
In progress — Write access still paused, but all proceeding according to plan. As soon as we have an ETA for completion we&#39;ll share it.
In progress — Write access to Chargebee, our billing system, is being paused now. All proceeding according to plan.
In progress — Maintenance started
Scheduled — During this maintenance window you won&#39;t be able to succesfully complete some operations such as:
<ul>
<li>make changes to your plan (buying a plan, upgrading / downgrading a plan, etc.)</li>
<li>update your billing profile</li>
<li>pay overdue amounts</li>
<li>destroy your account, if you are currently on a paid plan</li>
<li>transfer a site, if you are on a legacy &quot;per-project&quot; paid plan</li>
<li>destroy a site, if you are on a legacy &quot;per-project&quot; paid plan</li>
</ul>
Automatic plan adjustments that we apply to your plan, such as adding an extra collaborator, will be executed after the maintenance.

Resolved — This haven been fixed again.
Investigating — We are currently investigating an issue with projects usage statistics and API success rate statistics
Resolved — The issue han been fixed
Investigating — We are currently investigating an issue with projects usage statistics and API success rate statistics

Problems with projects usage statistics

Resolved — The incident can be closed now.
Monitoring — We have identified the source of the problem and applied a fix. We are monitoring the situation at the moment.
Investigating — We are currently investigating an high latency on some site-api.datocms.com endpoints requests

High latency on some network requests

Resolved — Since November 23th, due to a problem, the daily reports visible in the &quot;Project usages&quot; pages of your DatoCMS project did not get reset at about midnight (UTC time) as usual. The conseguence is that you might have noticed that the daily reports showed bigger numbers each day.
These reports includes data such as:
<ul>
<li>Top assets by traffic</li>
<li>Top API tokens for CDA</li>
</ul>
and they are not related to usage data we use to bill extra plan usages.
Today, we have fixed the origin problem and we have reset the daily reports. All existing data have been saved in the &quot;Current Month&quot; report, so no data has been lost.
Sorry for any problems this might have caused to you.

[Retrospective] Usage report statistics did not reset on daily basis

Completed — Everything looks good :)
Verifying — Maintenance done. We are perfoming some checks now.
In progress — Maintenance is in progress. Downtime is expected.
Scheduled — Maintenance will be executed at scheduled time.
Scheduled — We&#39;re going to upgrade our databases with a follower changeover strategy.
During this period of time the Content Management API, Content Delivery API and projects administrative interface will be completely unavailable.
Graphql requests requested in our CDN will continue to work.
The actual maintenance impact should be contained to 15 minutes starting from 2022-11-12T16:00:00Z (UTC).

Resolved — Everything is back to normal
Identified — The problem was caused by an abnormal amount of requests coming from a project.
The burst of requests is now ended so everything should be OK. We continue to investigate.
Investigating — We are experiencing high latency issue on our CMA/CDA endpoint. Requests might take longer to respond or fail after 30 seconds.
Admin interface might not load.
We are investigating.

High lantency on API requests

Resolved — unpkg.com is back online again
Identified — Due to a JS package CDN provider being down at the moment, some plugins cannot be loaded into DatoCMS admin interface.
This does not affect REST and GraphQL API requests.

Errors while loading plugins in admin interface

Resolved — The upstream provider maked the incident as resolved.
Monitoring — This should be fixed now. Upstream provider did not update their status page yet though
Identified — The upstream provider is working on a fix
Identified — It is confirmed that there is an issue with the upstream DNS provider (Heroku).
We will update this page as soon as we have news.
Cached graphql queries should not be affected by this down.
Investigating — We are still investigating.
It seems like an issue with an upstream DNS provider ( <a href="https://status.heroku.com/incidents/2453">https://status.heroku.com/incidents/2453</a> )
Investigating — We are currently investigating a problem while accessing DatoCMS projects and performing API calls to our CMA / CDA.

Problems while accessing DatoCMS projects and performing API calls

Resolved — The issue is closed.
Monitoring — Response times are back to normal. We are monitoring the situation.
Identified — The cause of the slowdowns is a single project configured to use an SSO provider that is unreachable at this time. We are releasing a fix to mitigate the problem.
Identified — We have identified the cause of a general slowdown in response times.

Slow response times

Resolved — The incident has been resolved.
Monitoring — A fix has been implemented by our partner and they are monitoring the result.
Identified — One of our partner (Pusher) is currently having an issue.
This could prevent our admin interface to show a notification for actions such as: create a new field, update a field, destroy a field, forking an enviroment, duplicating a site, uploading a new upload and so on.
Please, refresh your browser page as the underneath operation might have been completed successfully even if no notification is shown.
Follow this issue ( <a href="https://status.pusher.com/incidents/k8bxxnb3z35y">https://status.pusher.com/incidents/k8bxxnb3z35y</a> ) to get the most recent updates.
Also, this could lead to some builds failing with this message:
<blockquote>
Could not subscribe to real-time events!
</blockquote>
Sorry for the inconvenience

Problems with notifications on CMS and builds

Resolved — We have removed the temporary hotfix.
Due to this hotfix, project usages between 2022-07-28T16:00Z and 2022-07-29T07:30Z have not been saved.
Monitoring — Problems with the upstream provider has been fixed. No more customers reported this problem. Tomorrow we are going to remove the temporary fix we implemented earlier.
Identified — Failing builds on netlify could be related to this issue: <a href="https://www.netlifystatus.com/incidents/zspkr7fmdhvd">https://www.netlifystatus.com/incidents/zspkr7fmdhvd</a>
Identified — It looks like a problem within our cloudflare workers, but only in some regions of U.S.A.
We have found a temporary way to overcome the issue, while we are investigating more.
Investigating — Customers are reporting issues while performing requests to our CMA. We are looking into this.

Errors while performing requests to our CMA

Resolved — All streaming times have been successfully imported.
Monitoring — Today we have found a bug that prevented us from saving videos&#39; streaming times since July 11th for all projects.
We have fixed the bug and now we are backfilling the streaming times.

Missing streamed video times

Resolved — This incident has been resolved.
Monitoring — They report: &quot;A fix has been implemented and we are monitoring the results.&quot;
Identified — They are applying a fix.
Identified — Customers are reporting issues accessing their DatoCMS admin panel and accessing our CMA and CDA.
It looks like a major Cloudflare incident affecting some regions.
Please, follow this status to get real time updates: <a href="https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj">https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj</a>

Issues accessing DatoCMS APIs

Resolved — Digital ocean marked the issue as fully resolved now.
Monitoring — It looks like everything is back to normal right now, but Digital Ocean is still investigating on this. If this happens again, please use your default subdomain to access your admin project interface.
Identified — Digital Ocean is reporting multiple product failures at the moment: <a href="https://status.digitalocean.com/incidents/xkpsn9hbnnlz">https://status.digitalocean.com/incidents/xkpsn9hbnnlz</a>
Please, follow the above link to get updates.
Investigating — There are problems accessing DatoCMS projects via custom domains.
We are going to investigate this, but in the meantime please use your default DatoCMS subdomain (<code>&lt;your-project-subdomain&gt;.admin.datocms.com</code>) to access it. If you do not remember your project subdomain, you can always check it out in our <a href="https://dashboard.datocms.com">https://dashboard.datocms.com</a>

Cannot access admin interface via custom domain

Resolved — IMGIX completely fixed the service, restoring normal behaviour.
Identified — IMGIX reports elevated rendering errors, so images coming from <code>www.datocms-assets.com</code> might return errors.
Please, follow this incident to get updates: <a href="https://status.imgix.com/incidents/d9wr5vtpnbnp">https://status.imgix.com/incidents/d9wr5vtpnbnp</a>

Errors with image rendering

Resolved — Fastly reports the incidents as fixed
Identified — Our graphql CDN provider (Fastly) is reporting an issue with the purging system: <a href="https://status.fastly.com/incidents/7pps2nn0r9q0">https://status.fastly.com/incidents/7pps2nn0r9q0</a>

Stale graphql responses from CDA

Completed — POSTPONED
Our provider postponed the migration, so we are waiting for a new time window from them.
We&#39;ll create a new scheduled maintenance event here once we have more information.
Scheduled — On May 14th at 05:00 AM UTC we are going to perform a scheduled maintenance on our billing infrastructure. During this window (projected at 4-6 hours), project owners won&#39;t be able to successfully perform any billing related action, such as - but not limited to - plan change, credit card update, billing address update, project transfer, project deletion and account deletion.
Also, automatic plan adjustments will be delayed until the end of the maintenance.

Scheduled maintenance on billing infrastructure

Completed — Everything is nominal
Verifying — Maintenance completed. Monitoring
In progress — Maintenance in progress
Scheduled — We are ready to perform the maintenance at the scheduled time.
Scheduled — During this period of time the Content Management API, Content Delivery API and projects administrative interface will be completely unavailable. The actual maintenance impact should be contained to 10 minutes.

Scheduled database maintenance

Resolved — Build triggers and webhooks are now being processed in two separate queues, so that they do not affect each other anymore.
Identified — The source of the latency was traced down to an abnormal increase of webhook calls in that time frame.
Investigating — Between 2021-11-23T21:30:00+00:00 and 2021-11-23T23:20:00+00:00 there was an high increase of latency in the queue that process build triggers and webhooks.
This means that deploys and webhooks took more time than usual to being triggered.

Build trigger and webhooks delay (retrospective)

Resolved — Normal behaviour is fully restored, resolving the ticket.
Monitoring — We performed a database upgrade to make room for the increased number of requests. Most timeout errors are gone, and the system is catching up. We&#39;ll continue to monitor the situation.
Investigating — We&#39;re seing a number of CMA/CDA requests failing. Seems to be a DB overload, we&#39;re investigating the root cause, and will keep on updating this ticket.

Requests slow/failing

Resolved — The issue is resolved
Monitoring — We are monitoring
Monitoring — Rollback completed. Please, refresh your browser if you still get the error window.
Identified — A bug in the latest deploy caused this. We are rolling back it
Investigating — We got reports of problems occurring while trying to visiting or saving some kind of records. We are investigating

Problems when reading / saving some kind of records through our admin interface

Resolved — Heroku posted a post-mortem update: <a href="https://status.heroku.com/incidents/2359">https://status.heroku.com/incidents/2359</a>
Resolved — <code>webhooks.datocms.com</code> is working as expected now
If you see some builds stuck in the &quot;build in progress&quot; stage, please mark them as failed and try again
Monitoring — All domains are pointing to the old Heroku application again, but we are still having issues with <code>webhooks.datocms.com</code>. We are investigating.
Monitoring — Heroku is solving the root issue. We&#39;re waiting until the incident is fully resolved on their end to switch back to our original application.
Monitoring — We have created a new application and changed DNS entries to circumvent Heroku&#39;s issues, and it seems to be working: requests are coming to our application again. We&#39;re monitoring the situation of course, but things should be back to normal.
Investigating — We are applying a mitigation strategy in the meantime
Investigating — Heroku is investigating an issue on their platform ( <a href="https://status.heroku.com/incidents/2359">https://status.heroku.com/incidents/2359</a> )
Investigating — We are experiencing some connectivity issues at the moment.
Investigation is in progress.

Connectivity issues

Resolved — We&#39;ve talked with Vercel, and they&#39;ve told us that the only way to make notifications work again is by redoing the integration process. To do that, enter the failing Build Trigger in DatoCMS and press the &quot;Authorize Vercel&quot; button again.
Identified — Vercel has removed the old way of notifying third-party (that&#39;s us) about deployment status (build success/error). We&#39;re working on a solution.

Vercel build notifications not working

Resolved — Everything is back to normal
Monitoring — We are still monitoring
Identified — A mitigation strategy is in progress
Investigating — We are seeing an increase of timed out requests since 2021-08-19T01:51:53Z

High number of failing requests

Resolved — Everything is OK now. We&#39;ll update this later if one of our hosting partners writes down something on their status pages.
Monitoring — Requests are back to normal without our intervention.
We are monitoring the situation.
Investigating — We&#39;re seing a number of CMA requests failing — some of them with CORS issues. We&#39;re investigating the cause, no new code has been deployed in this regard.

Slow responses on CMA

Resolved — We are closing this issue for now
Monitoring — A problem with DNS resolution has been fixed and now it should be back to normal.
We are monitoring the situation
Investigating — We have received reports of problems accessing admin UI through the custom domains.
Accessing admin UI through its default subdomain ( <code>&lt;project&gt;.admin.datocms.com </code>) is still possible.

Problem accessing admin interface though custom domains

Resolved — The incident has been resolved.
Monitoring — We&#39;ve removed Fastly altogheter and put plan B into action: tripled the resources and fallback to custom cache on Redis. We&#39;re back online and running again. We are monitoring requests to see how they progress over time.
Monitoring — Even Netlify CI is having issues: please follow this Twitter thread for live updates about the incident.
<a href="https://twitter.com/datocms/status/1402204141259214850">https://twitter.com/datocms/status/1402204141259214850</a>
Investigating — We&#39;re investigating the cause, seems to be related to Fastly (their status page and homepage are currently down).

500 status for Content Delivery API

Resolved — Closing the issue as resolved
Monitoring — The situation is back to normal again. We&#39;re sorry for any the inconvenience, and we&#39;re continuing to analyze the traffic to ensure that system performance continues to remain consistent.
Monitoring — Regular rate limits have been restored.
Monitoring — The reason for the problems concerns a sudden spike in the number of concurrent requests. The situation has returned to more or less normal levels, but we temporarily halved the API request rate limits (from 40 reqs/secs -&gt; 20 reqs/sec) to provide better service to the greatest number of customers while the situation stabilizes. We will restore regular rate limits shortly.
Investigating — We mitigated the issue, error rate has dramatically lowered but some requests are still returning errors. We are continuing to analyze the traffic.
Investigating — We’re receiving feedback of slow responses and intermittent 503 error codes on our Content Management API, we’ve doubled up our server resources and investigating in the meantime

Slow Responses And Intermittent 503 Error Codes

Resolved — Closing the issue.
Monitoring — We got reports of webhooks and build trigger being stuck or slow.
The queue latency is now back to normal. We are monitoring the situation. Sorry for the inconvenience.

Slow webhooks calls / build triggers events

Resolved — No more reports about this error so far. We are closing this.
Monitoring — We got reports of errors <code>genericError.MISSING_FIELDS</code> when trying to save an item that contains link or assets fields inside modular blocks.
We deployed a fix and we are monitoring right now.
This does not effect the CMA, only the admin interface.

Cannot save items that contain some kind of fields in modular blocks (`genericError.MISSING_FIELDS`)

Completed — Done.
Scheduled — To improve the scalability of the service, starting from 11th November webhooks and build triggers requests will be performed by dynamic IPs.
If you have a particular need, please contact support: <a href="https://www.datocms.com/support">https://www.datocms.com/support</a>

IP addresses of webhook calls and build triggers will change

Completed — Everything is nominal. Maintenance completed!
Verifying — Maintenance completed. Monitoring.
In progress — Maintenance started.
In progress — Maintenance is ready to be performed.
Scheduled — UPDATED: Maintenance postponed to 11th October.
We&#39;re going to upgrade our database with a follower changeover strategy.
During this period of time the Content Management API, Content Delivery API and projects administrative interface will be completely unavailable. The actual maintenance impact should be contained to 10 minutes.

Scheduled database maintenance [UPDATED]

Resolved — CMS interface has been unavailable for around 6 hours from around 12AM to 6AM UTC on the 29th of September.
All APIs were up and running, meaning that all the sites that were using our APIs were unaffected.
We have identified the root cause of the issue in a our Nginx + auto SSL configuration. We use auto SSL (a Lua plugin for Nginx) to issue certificates for new custom CMS addresses (like datocms.yourdomain.com). At the beginning of the issue a new site has been created with a custom domain that wasn&#39;t properly configured. This caused auto SSL to fail and try different connections using IPv6, which wasn&#39;t configured on the server. This caused Nginx to stop responding even if the service was up. So on server restart everything worked as expected.
We have now added an IPv6 address to the machine to prevent this from happening again.
Also our status page is showing the downtime of the issue and we are putting in place systems to handle these type of events in a quicker manner.

CMS interface unreachable

Resolved — This incident has been resolved.
Monitoring — Everything should be back to normal now. We are monitoring.
Investigating — We are investigating an error accessing our CDA endpoint (graphql.datocms.com).
This does not affect neither our CMA endpoint (site-api.datocms.com), nor DatoCMS admin interface.

Errors while accessing CDA (graphql.datocms.com)

Resolved — Closing this issue
Monitoring — Even the last bug on records version history has been resolved. We are now monitoring.
Identified — A fix for the previous problems has been deployed. We are working on another problem regarding records version history.
Identified — We got reports about problems while saving alt/title in Media area, SEO settings, roles, build triggers and webhooks.
This does not affect our REST API.
We are working on a fix

Problems while saving some type of resources in Admin interface

Resolved — Our CDN provider noted a performance issue with cache purging between Aug 24, 2020 - 23:10 UTC and Aug 25, 2020 - 02:44 UTC.
This may have caused a slowdown in purging the Graphql query cache.
<a href="https://status.fastly.com/incidents/hyj3fw6vyggm">https://status.fastly.com/incidents/hyj3fw6vyggm</a>

Retrospective - Performance issues with Graphql cache purging

Resolved — No more errors arosed. Closing this incident.
Monitoring — We are investigating the cause of a elevated error rate occurred between 09:09 UTC and 10:30 UTC today while accessing CMA.
The requests are now back to normal and we are monitoring.

Elevated error rate while accessing CMA

Resolved — Closed.
Monitoring — It should be fixed by now. We will keep monitoring
Investigating — There is an issue with the statistics collector for the &quot;Content Delivery API response time&quot; graph.

Issue with some status page data

Resolved — Cloudflare marked the issue as resolved
Identified — <blockquote>
Cloudflare is investigating issues with Cloudflare Resolver and our edge network in certain locations.
</blockquote>
<a href="https://www.cloudflarestatus.com/incidents/b888fyhbygb8">https://www.cloudflarestatus.com/incidents/b888fyhbygb8</a>

Cloudflare Network and Resolver Issues

Completed — Maintenance completed.
Scheduled — On Saturday, June 27, 2020, our transactional email provider will be performing necessary database maintenance starting at 10:00 am Eastern Time. They expect the maintenance to last about 2 hours.
All outbound messages will be queued for approximately 10-15 minutes during the maintenance.

Transactional email service maintenance

Completed — Maintenance completed on our DB. No downtime noticed!
Scheduled — Our DB needs maintenance from Heroku.
During our maintenance window 2020-05-24 00:00:00 - 04:00 UTC Heroku will create a hidden DB follower, wait for it to catch up, swap the DB, unfollow it from our former leader and repoint any followers to properly follow our new leader. This should not cause any downtime.

Database maintenance

Resolved — Everything is back to normal right now
Monitoring — The fix in the API seems to be working as expected, we are monitoring the situation, which should be back to normal.
If possible please refresh the CMS pages that you have open as that should also help.
Identified — We have found an issue with record validation that is being triggered more often than necessary slowing everything down. We are throttling the validations and things seems to be back under control. We&#39;ve also issued a fix on the CMS that should help.
Investigating — We have found different potential problems, we are trying some changes to see if the response time improves.
Investigating — We are investigating slow response times in CMA and CDA.

Slow response times in APIs

Completed — Update done.
Scheduled — We&#39;ll release an update to the CDA that will cause some changes on filtering.
Read all the details in the documentation: <a href="https://www.datocms.com/docs/content-delivery-api/changelog#20200511---changes-in-graphql-filtering">https://www.datocms.com/docs/content-delivery-api/changelog#20200511---changes-in-graphql-filtering</a>

Changes in GraphQL filtering

Resolved — The issue is now resolved.
Identified — The issue has been identified and the resolution is in progress.
Identified — The Imgix team has put in place a mitigation for the problem. Keep following on their page: <a href="https://status.imgix.com/incidents/n77l9lgsk15k">https://status.imgix.com/incidents/n77l9lgsk15k</a>
Investigating — We are experiencing an elevated error rate on image rendering from our partner Imgix.
They have a case open on their status page that you can follow: <a href="https://status.imgix.com/incidents/n77l9lgsk15k">https://status.imgix.com/incidents/n77l9lgsk15k</a>

Rendering errors on assets

Completed — Everything is ok. Ending monitoring.
Verifying — Upgrade and migration completed. We are monitoring. In the meanwhile the system is back online.
In progress — Still going
In progress — Start
Scheduled — Preparing for maintenance
Scheduled — We&#39;re going to upgrade our database and, meanwhile, we will deploy some migrations.
All DatoCMS services will be temporarily put in maintenance mode (~ 5 minutes).

Scheduled database upgrade

Completed — Completed verifying, closing the case
Verifying — Migration completed! We&#39;re verifying everything went as expected, but meanwhile the system is up again.
In progress — Migration started!
Scheduled — We&#39;re going to deploy some migrations on our database that will require our REST API to be temporarily put in maintenance mode (3 minutes).

Scheduled database migration

Completed — Closing as everything seems to be working correctly.
Verifying — The migration has finished successfully, we&#39;ll be monitoring the situation for a few more hours.
In progress — We are starting the migration with some delay now.
Scheduled — dashboard.datocms.com will be migrated to a new location and might be unreachable for a few minutes

Dashboard maintenance

Completed — Everything is completed and we didn&#39;t experience any significant outage.
Scheduled — Our infrastructure provider Digital Ocean, is going to perform an upgrade of core networking equipment to improve performance and reliability for our Frankfurt (Germany) data center.
Expected Impact: 
There will be several minutes of packet loss while they shift traffic between redundant devices. There may be additional event processing delays.
Service impacted: admin interface, Community, site search, timing reports on status page.

Network Maintenance in data center

Resolved — Everything is back to normal. CDA response times are still higher than usual and we&#39;ve tracked down the cause. We&#39;ll improve the situation in the next hours.
Monitoring — We have upgraded our DB instance as we are seeing an unusual spike of very complex queries that were slowing down the system. The system is now operational again, we&#39;re monitoring the situation.
Investigating — We are seeing a low percentage of errors and slow performance both on our CDA and CMA. We&#39;re currently investigating

Elevated error rates

Completed — Everything is working as expected, no issue has been found. Closing.
Verifying — Community migration has completed successfully.
We&#39;ll be monitoring the situation for a few hours.
In progress — Site search has been migrated successfully. Scraping and search functionality should be working as normal. Scraping that were launched during the migration has been marked as failed and might need redoing.
Community migration is still in progress.
In progress — We have taken down Community and Site search to snapshot and move the services.
Scheduled — We need to migrate Community and Site Search to new machines and services might be unavailable for up to 3 hours Friday 27th December 2019 from 10AM CET.

Community and Site Search unavailable

Completed — The maintenance took 3 minutes and we didn&#39;t observe any incident at the time. Excellent job Heroku!
Scheduled — During our maintenance window 2019-11-10 00:00 - 04:00 UTC Heroku will create a hidden DB follower, wait for it to catch up, swap the DB, unfollow it from our former leader and repoint any followers to properly follow our new leader.
This should not cause any downtime.
Scheduled — Our DB needs maintenance from Heroku.
During our maintenance window 2019-11-10 00:00 - 04:00 UTC Heroku will create a hidden DB follower, wait for it to catch up, swap the DB, unfollow it from our former leader and repoint any followers to properly follow our new leader. This should not cause any downtime.

Resolved — Both rendering times and rendering errors have return to normal.
Monitoring — Imgix has rolled out a fix and is monitoring the situation which should be back to normal for most users: <a href="https://status.imgix.com/incidents/bg0msyrz9f17">https://status.imgix.com/incidents/bg0msyrz9f17</a>
Identified — Imgix is our service provider for asset caching and manipulation. They are seeing errors that are affecting our asset uploads and serving. Read all the details on their status page: <a href="https://status.imgix.com/incidents/bg0msyrz9f17">https://status.imgix.com/incidents/bg0msyrz9f17</a>

[Imgix] Elevated rendering errors

Completed — All the systems are back online, sorry for the inconvenience. Enjoy your new Media Area! :)
Scheduled — We are going to perform a maintenance task on our master database and we expect all our APIs to be unavailable for around 15 minutes.
This task is going to be necessary to prepare for the upcoming changes in the media area.

Resolved — We have identified and improved the query that has caused a snowball error in our database bringing most of the queries to timeouts.
The total number of record versions saved in our DB is growing fast and it&#39;s causing new challenges for us. We&#39;ll monitor the situation in the next few days. Hopefully we&#39;ll be faster next time as we have also improved our error reporting from the DB.
Identified — We have found the root cause in a slow query that was bringing down the database for everyone.
We have limited the query duration and the status seems back to normal.
We are monitoring and trying to release a stable fix.
Investigating — We are seeing an increased number of calls timing out.
When we see a spike of timeouts we can see errors across different endpoints.
We are investigating the root cause of the issue and we&#39;ll report back as soon as possible.

Increased number of timeouts

Resolved — Two outages happened together on 24th of June 2019 impacting different areas of DatoCMS.
One was Cloudflare: <a href="https://www.cloudflarestatus.com/incidents/46z55mdhg0t5">https://www.cloudflarestatus.com/incidents/46z55mdhg0t5</a> which would have impacted the GraphQL endpoint which had given time outs to certain IP ranges.
The other was with Heroku, which have impacted both GraphQL and REST APIs: <a href="https://status.heroku.com/incidents/1804">https://status.heroku.com/incidents/1804</a>
If you want more information or leave a comment, you can do that here: <a href="https://community.datocms.com/t/downtime-june-24th-2019/53">https://community.datocms.com/t/downtime-june-24th-2019/53</a>

Network connection issues

Resolved — The issue is now resolved.
Investigating — Imgix is currently having some troubles returning images in some particular regions, we&#39;re investigating. For more information: <a href="https://status.imgix.com/">https://status.imgix.com/</a>

Imgix Degraded Performance

Completed — We made it! We received a bunch of reports of weird things happening in some edge-cases, but overall it was a pretty smooth migration. Thanks for your help!
Scheduled — We&#39;re going to deploy some big internal code refactorings to the React app that powers the DatoCMS administrative areas. We&#39;ll follow the deploy closely, but if you happen to find any kind of annoyance/bug, please contact us via <a href="mailto:support@datocms.com">support@datocms.com</a> or chat.

Big upgrade of React codebase

Resolved — This issue is now resolved.
Monitoring — Heroku has pushed a fix and we are seeing error rates decrease. We will continue monitoring to ensure that this issue is fully resolved.
Investigating — We are investigating an issue on our provider (Heroku) that is causing instability in managing applications. You may experience routing errors contacting both our CDA ad CMA.

Routing errors

Resolved — This issue is now resolved.
Monitoring — The problem seems to have disappeared, we&#39;re monitoring the situation.
Investigating — You may experience some errors contacting both our CDA ad CMA, a lot of request are failing due to timeout. Investingating.

Request Timeout

Completed — Maintenance was successfully completed!
Scheduled — We&#39;re performing database maintenance at this time. During the maintenance window, parts DatoCMS may be sporadically unavailable for a small percentage of customers.

Scheduled Database Maintenance

Resolved — The problem is now resolved
Monitoring — We identified the source of the problem and shipped a fix. We&#39;ll continue monitoring the situation
Investigating — We are seeing a low level of 5xx errors for some Delivery API traffic, we are investigating the issue.

Is DatoCMS down?

DatoCMS status is UP

Lastest published events

Issues sending emails to customers

🛠️ Scheduled database maintenance [ ⚠️ READ-ONLY MODE ]

Imgix partial outage and rendering errors

DatoCMS Services, Components and Regions

Recent DatoCMS events

Issues sending emails to customers

🛠️ Scheduled database maintenance [ ⚠️ READ-ONLY MODE ]

Imgix partial outage and rendering errors

Issues sending emails to customers

Increase in error rate and response time

Asset uploads temporarily unavailable