Jun <var data-var='date'>15</var>, <var data-var='time'>14:14</var> UTC Resolved - The MDS API is now operational. Jun <var data-var='date'>15</var>, <var data-var='time'>13:29</var> UTC Identified - Requests to the MDS API are returning errors. We've identified the issue and are working on a fix.

MDS API outage

Mar <var data-var='date'>10</var>, <var data-var='time'>20:03</var> UTC Resolved - We have identified and resolved the issue. New records and updates should now appear as expected in services like Fabrica, Commons, and the REST API, and queued changes have been processed. Mar <var data-var='date'>10</var>, <var data-var='time'>19:16</var> UTC Investigating - We are investigating an issue where new DOI records and DOI record updates may be taking longer than usual to appear in services like Fabrica, Commons, and the REST API. API requests are still being received and processed successfully.

Delayed indexing

Dec <var data-var='date'>17</var>, <var data-var='time'>23:47</var> UTC Resolved - Performance of all services has returned to normal levels. Dec <var data-var='date'>17</var>, <var data-var='time'>17:08</var> UTC Monitoring - Performance has stabilized. We are continuing to monitor. Dec <var data-var='date'>17</var>, <var data-var='time'>16:26</var> UTC Investigating - We are investigating degraded performance across our services. This is manifesting as occasional error messages or slow response times for API requests and/or frontend services.

Service instability and occasional errors

Dec <var data-var='date'>16</var>, <var data-var='time'>09:59</var> UTC Completed - All services are operational and no further problems have occurred. Dec <var data-var='date'>16</var>, <var data-var='time'>08:59</var> UTC Verifying - The database upgrade is complete, but we experienced some problems with DNS caching the older database server address. We are continuing to verify the current status. Dec <var data-var='date'>16</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>11</var>, <var data-var='time'>13:09</var> UTC Scheduled - We are performing a routine database version upgrade for our production system, this is for on-going support and stability. We will attempt to minimise any downtime during this period for the affected services. We are performing a blue/green deployment and therefore any outage should be very minimal, within minutes or less.

Database Upgrade - Production

Dec <var data-var='date'>12</var>, <var data-var='time'>08:55</var> UTC Completed - The scheduled maintenance has been completed. Dec <var data-var='date'>12</var>, <var data-var='time'>08:22</var> UTC Verifying - Upgrade has been completed, we're just confirming everything is operational. Dec <var data-var='date'>12</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>11</var>, <var data-var='time'>13:04</var> UTC Scheduled - We are performing a routine database version upgrade for our test system, this is for on-going support and stability. We will attempt to minimise downtime during this period, but the test system and associated services may be briefly unavailable.

Database upgrade - Test

Oct <var data-var='date'> 9</var>, <var data-var='time'>12:58</var> UTC Resolved - Intermittent performance issues have stabilized. We will continue to monitor. Oct <var data-var='date'> 8</var>, <var data-var='time'>17:51</var> UTC Investigating - We are investigating intermittent performance issues affecting our services, which may result in slow response times for API requests and frontend applications.

Intermittent performance issues

Oct <var data-var='date'> 7</var>, <var data-var='time'>15:09</var> UTC Resolved - Performance of all services has returned to normal levels. Oct <var data-var='date'> 7</var>, <var data-var='time'>14:35</var> UTC Monitoring - Our service autoscaling increased resources to handle a higher amount of traffic. This has improved performance, and we are continuing to monitor the situation. Oct <var data-var='date'> 7</var>, <var data-var='time'>13:53</var> UTC Investigating - We are investigating degraded performance across our services. This is manifesting as occasional error messages or slow response times for API requests and/or frontend services.

Aug <var data-var='date'>21</var>, <var data-var='time'>05:11</var> UTC Resolved - All systems are operational and data unavailable in search index has been restored. Aug <var data-var='date'>20</var>, <var data-var='time'>14:47</var> UTC Monitoring - We've added a further fix and all search indexing and related services should be operational. We are now monitoring and looking to clean up any missing data in the indices. Aug <var data-var='date'>20</var>, <var data-var='time'>13:55</var> UTC Identified - A temporary fix has been put in place and things should be working now, but we may still have a period of time where some things have not been indexed. We are working on a longer term fix for increased demand and also add any missing items (i.e. DOIs) to our search indices. Aug <var data-var='date'>20</var>, <var data-var='time'>12:44</var> UTC Investigating - We're experiencing some timeouts adding to our search index cluster, this will mostly be noticeable in DOIs that are registered might not be findable via certain API endpoints and search/management interfaces. We believe this is a disk space issue due to increased demand and are investigating a remedy. Note no actual data loss as this is just the indexing side, data storage is unaffected and therefore DOI registration is operational.

Indexing services outage

Jul <var data-var='date'>17</var>, <var data-var='time'>15:21</var> UTC Resolved - The DataCite homepage at https://datacite.org and the Make Data Count homepage at https://makedatacount.org are now available. We will continue to monitor. Jul <var data-var='date'>17</var>, <var data-var='time'>13:55</var> UTC Identified - We've identified an issue that may affect access to the DataCite homepage at https://datacite.org and the Make Data Count homepage at https://makedatacount.org. Other DataCite services and APIs are not affected. The issue has been isolated, and we are working on a resolution.

DataCite homepage temporarily unavailable

Jul <var data-var='date'> 8</var>, <var data-var='time'>16:58</var> UTC Resolved - Test services are restored. Jul <var data-var='date'> 8</var>, <var data-var='time'>16:37</var> UTC Identified - The test services that are powered by our OpenSearch cluster are currently not accepting new index writes. This means data is still being saved to the test database, but new records and record updates (for DOIs, members, repositories, etc.) are not appearing in REST API list responses, Fabrica list views, and other places that rely on the OpenSearch index We believe we already know the issue but are currently discussing about how best to resolve. Production services are not affected and are operating as normal.

Test DOI Indexing and related services is unavailable on test systems

Jun <var data-var='date'>18</var>, <var data-var='time'>11:33</var> UTC Resolved - Around 09:30 UTC to 10:00 UTC we had a minor outage within part of our infrastructure that processes various messages on queues (AWS SQS). This mostly meant that messages that usually processes tasks like indexing into our search cluster may have gone missing, this namely would affect things like DOI updates appearing in Commons/Fabrica public APIs. No data loss occurred as all data is saved to our database before background messages are sent to queues for processing. Users may have noticed http 500 (Internal Server Error) responses from our API endpoints during this period of time, but again the data was saved for any DOI create/update requests, it is also safe to re-run any requests that received these errors. The main issue that has arisen is some inconsistencies for a small amount of DOIs in our search index and possibly some URL updates to the global handle service that haven't gone through yet. We are working to clean up any inconsistencies, this should be complete within the next few hours.

Minor outage of background message systems

Jan <var data-var='date'>24</var>, <var data-var='time'>15:48</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>24</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>21</var>, <var data-var='time'>14:54</var> UTC Scheduled - After upgrades previous week this maintenance period is to change the server instance type to newer server hardware. This will be a blue/green deploy and no outages are expected, possible search timeouts might occur within brief 1-2 minute time window during switchover.

Search cluster server instance upgrade

Jan <var data-var='date'>17</var>, <var data-var='time'>16:12</var> UTC Completed - Primary updates to search cluster versions were completed. Jan <var data-var='date'>17</var>, <var data-var='time'>12:48</var> UTC Update - We're still in the process of updating services, it is taking longer than hoped, but no disruption is expected. Jan <var data-var='date'>17</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>16</var>, <var data-var='time'>13:38</var> UTC Scheduled - Routine upgrades to our OpenSearch production cluster. This is an in-place upgrade and no downtime is expected.

Opensearch routine maintenance upgrades

Dec <var data-var='date'>17</var>, <var data-var='time'>12:54</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'>17</var>, <var data-var='time'>09:56</var> UTC Monitoring - Mitigation strategies have been deployed to help with some anonymous traffic that was causing increased load on our search cluster. Services appear to be resuming normal function but with potentially still some degraded performance, we will continue monitoring. Dec <var data-var='date'>16</var>, <var data-var='time'>19:28</var> UTC Investigating - We're currently experiencing increased load on our services and are investigating the underlying cause and mitigation that we can take. This mostly appears to be affecting our search cluster, and therefore has a knock-on effect to queries and retrieval through DataCite Commons and some functionality in Fabrica. DOI Registration services should be unaffected, although may experience timeouts viewing content after registration.

Service slowness and occasional errors

Nov <var data-var='date'> 1</var>, <var data-var='time'>06:20</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>31</var>, <var data-var='time'>17:21</var> UTC Monitoring - Services appear stable, we will continue monitoring. Oct <var data-var='date'>31</var>, <var data-var='time'>16:47</var> UTC Update - We are continuing to investigate this issue. Oct <var data-var='date'>31</var>, <var data-var='time'>15:55</var> UTC Investigating - We're currently experiencing increased load on our services, we're investigating mitigation strategies. This mostly appears to be affecting our search cluster. Registration services should be unaffected, though may experience timeouts viewing content through frontend services / api search interfaces.

Service slowness

Oct <var data-var='date'>31</var>, <var data-var='time'>08:58</var> UTC Resolved - Yesterday we made some changes to how traffic was handled by our services, this combined with overall load reducing, everything should be working as normal now. Oct <var data-var='date'>30</var>, <var data-var='time'>12:27</var> UTC Investigating - We've noticed an increase load on our services which is having an affect on returning search results in different places. We are investigating possible causes and attempting to compensate.

Search cluster slow affecting multiple services.

Oct <var data-var='date'>17</var>, <var data-var='time'>12:00</var> UTC Completed - The scheduled maintenance has been completed. Oct <var data-var='date'>17</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>16</var>, <var data-var='time'>16:48</var> UTC Scheduled - We will be performing an upgrade of our production database servers. This is to ensure long term maintainability and stability. During this time there may be a brief outage to DOI registration services.

Production Database Upgrade

Sep <var data-var='date'>30</var>, <var data-var='time'>07:40</var> UTC Completed - The scheduled maintenance has been completed. Sep <var data-var='date'>30</var>, <var data-var='time'>07:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>26</var>, <var data-var='time'>15:40</var> UTC Scheduled - We will be performing essential maintenance to our version of the Firewall service provided by AWS. During this maintenance window we will be making the switch over to the new version, this should be a matter of minutes and no impact is expected.

AWS Firewall Upgrade

Jul <var data-var='date'>15</var>, <var data-var='time'>13:57</var> UTC Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>15</var>, <var data-var='time'>06:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>12</var>, <var data-var='time'>14:52</var> UTC Scheduled - Time is needed to complete the full upgrade from Elasticsearch to the latest Opensearch version (see previous Maintenance). As before no impact should be noticed and DOI registration will not be affected during this window.

Opensearch latest version upgrade

Jul <var data-var='date'>12</var>, <var data-var='time'>14:49</var> UTC Completed - We have upgraded in part the service to an intermediary version of Opensearch. This took longer than expected, to complete upgrade to the latest version, there will be another scheduled maintenance for the coming Monday. Jul <var data-var='date'>12</var>, <var data-var='time'>10:31</var> UTC Update - The scheduled maintenance is taking longer than anticipated, however still going as intended. The window has been extended for the day. No impact should be currently noticeable. Jul <var data-var='date'>12</var>, <var data-var='time'>07:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>10</var>, <var data-var='time'>16:32</var> UTC Scheduled - We will be undergoing an upgrade/migration to Opensearch for our search based services. To ensure on-going support and security updates it is important we run a maintained and supported version of our search software. Opensearch is an open source fork of Elasticsearch 7 and has seen it's own improvements, more importantly we are able to easily migrate our existing AWS managed instances. This upgrade will be in a blue/green deployment strategy and no downtime is expected however there maybe a brief 1-2 minutes of connection issues on some services as the switch happens, but no DOI registration functionality will be affected.

Elasticsearch Migration to Opensearch

May <var data-var='date'>28</var>, <var data-var='time'>08:00</var> UTC Completed - The scheduled maintenance has been completed. May <var data-var='date'>28</var>, <var data-var='time'>07:53</var> UTC Verifying - Verification is currently underway for the maintenance items. May <var data-var='date'>28</var>, <var data-var='time'>07:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>24</var>, <var data-var='time'>15:46</var> UTC Scheduled - We will be performing an upgrade of our production database servers. This is to ensure long term maintainability and stability. During this time there may be a brief outage to DOI registration services.

Production database upgrade

Mar <var data-var='date'>14</var>, <var data-var='time'>18:05</var> UTC Resolved - All systems are operational and working. We continue to investigate whether there are any data issues and we will contact impacted members if necessary. Mar <var data-var='date'>13</var>, <var data-var='time'>16:02</var> UTC Monitoring - DataCite experienced issues with our database services earlier today. Systems are now operational and we continue to monitor closely. We are investigating whether DOI registration and/or metadata updates were impacted and if so, will contact affected members directly. Mar <var data-var='date'>13</var>, <var data-var='time'>14:36</var> UTC Identified - We have restored connectivity to the database, DOI registration and Search/Index lookups are now operational but marked as degraded while we establish further incident root causes, we will update further once we have a clear picture. Mar <var data-var='date'>13</var>, <var data-var='time'>14:15</var> UTC Update - We are continuing to investigate this issue. Mar <var data-var='date'>13</var>, <var data-var='time'>11:54</var> UTC Investigating - We are currently experiencing problems with our database services. This is affecting all DOI registration services and some search/indexing service responses. We are investigating as a matter of priority.

Services unavailable

Jan <var data-var='date'>10</var>, <var data-var='time'>09:44</var> UTC Completed - The upgrade completed successfully with no interruption to services. Jan <var data-var='date'> 9</var>, <var data-var='time'>19:01</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'> 9</var>, <var data-var='time'>17:23</var> UTC Scheduled - We are planning to upgrade the capacity of the ElasticSearch system underlying our API services. This will be a blue/green deployment and as such we don't anticipate any impact on our services during this procedure. DOI registrations, Fabrica, DataCite Commons, and all of our APIs will remain available during this time.

ElasticSearch upgrade

Jan <var data-var='date'> 8</var>, <var data-var='time'>09:51</var> UTC Resolved - Services are operational. We continue to investigate and consider further long term improvements. Jan <var data-var='date'> 5</var>, <var data-var='time'>14:33</var> UTC Monitoring - Yesterday evening (UTC) we introduced a few extra changes to increase support for load in certain areas of our services. We will continue to monitor the situation, services are still slightly degraded but overall should be functional. Jan <var data-var='date'> 4</var>, <var data-var='time'>17:16</var> UTC Identified - The issues currently seen are related to increased load across our services, some specific problems have been identified and resolved. Services are not down, but you may experience intermittent timeouts/errors depending on which service. we are looking at various options to further improve this and increase our capacity. Jan <var data-var='date'> 4</var>, <var data-var='time'>11:00</var> UTC Investigating - We are investigating degraded service across our services. This is manifesting as occasional error messages or slow response times for API requests and/or frontend services. This appears to be due to increased traffic across but we are investigating the source and what capacity we can increase.

Services experiencing reduced performance and instability.

Jan <var data-var='date'> 2</var>, <var data-var='time'>14:40</var> UTC Resolved - All systems remain operational, overall health of services appears normal. Incident Resolved. Jan <var data-var='date'> 2</var>, <var data-var='time'>10:51</var> UTC Monitoring - We have identified a problem and individual DOI detail pages should be available again within commons. We however have been receiving and influx in high usage from some automated sources, so we will continue to monitor for additional effects. Dec <var data-var='date'>28</var>, <var data-var='time'>11:00</var> UTC Update - We are continuing to investigate the underlying cause of the errors appearing on individual DOI pages on the DataCite Commons service. Individual DOI metadata can be accessed via REST and GraphQL - for more information, please see our Support Pages: https://support.datacite.org/docs/api and https://support.datacite.org/docs/datacite-graphql-api-guide Dec <var data-var='date'>27</var>, <var data-var='time'>12:25</var> UTC Investigating - We are aware of an issue where individual DOI pages on DataCite commons are returning a 404 Error and are investigating the root cause. Individual DOI metadata and related information can still be accessed via the REST and GraphQL API endpoints.

Is DataCite down?

DataCite status is UP

Lastest published events

MDS API outage

Delayed indexing

Service instability and occasional errors

DataCite Services, Components and Regions

Recent DataCite events

MDS API outage

Delayed indexing

Service instability and occasional errors

Database Upgrade - Production

Database upgrade - Test

Intermittent performance issues