Jul <var data-var='date'>27</var>, <var data-var='time'>19:21</var> UTC Resolved - Processing times are back to normal. This issue is now resolved. Post-mortem to follow soon. Jul <var data-var='date'>27</var>, <var data-var='time'>19:20</var> UTC Update - Processing time keeps improving, all systems are now operational. Jul <var data-var='date'>27</var>, <var data-var='time'>19:18</var> UTC Monitoring - We implemented a fix and continue to monitor the quality of processing time, which is now improved. Jul <var data-var='date'>27</var>, <var data-var='time'>18:51</var> UTC Identified - We've identified the root cause of degraded performance in our US search cluster and are actively working to resolve it. We'll follow up once fully restored. Jul <var data-var='date'>27</var>, <var data-var='time'>18:23</var> UTC Update - We are seeing degraded performance of some nodes in our search cluster for indexed faces. We continue to investigate the root cause and possible remediations. Jul <var data-var='date'>27</var>, <var data-var='time'>18:12</var> UTC Investigating - We are currently investigating delays in our US cluster for the processing of Facial Similarity, Document and Known Faces reports.

Delays in processing reports in the US cluster

Jul <var data-var='date'>24</var>, <var data-var='time'>07:08</var> UTC Resolved - The provider has resolved the issue, and the service is now operating normally Jul <var data-var='date'>24</var>, <var data-var='time'>05:09</var> UTC Monitoring - We are seeing our provider's service recovery and request success rates improving. We continue to monitor the situation closely. Jul <var data-var='date'>24</var>, <var data-var='time'>04:55</var> UTC Update - We are continuing to work on a fix for this issue. Jul <var data-var='date'>24</var>, <var data-var='time'>04:36</var> UTC Identified - We are working with our third-party provider to resolve the issue and will share updates as they become available. Jul <var data-var='date'>24</var>, <var data-var='time'>04:18</var> UTC Investigating - Identity Enhanced is experiencing a major decrease in clear rates for applicants all other the world except UK

Identity Enhanced issues

Jun <var data-var='date'>23</var>, <var data-var='time'>03:49</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>23</var>, <var data-var='time'>03:45</var> UTC Monitoring - The issue has been fixed. We're monitoring and resuming blocked QES tasks, if they can be. Jun <var data-var='date'>23</var>, <var data-var='time'>03:20</var> UTC Update - Our provider is still experiencing issues. All QES tasks cannot complete for now. Jun <var data-var='date'>23</var>, <var data-var='time'>02:57</var> UTC Identified - Our provider is experiencing issues. All QES tasks cannot complete for now. Jun <var data-var='date'>23</var>, <var data-var='time'>02:38</var> UTC Investigating - We cannot complete QES tasks because our electronic signature provider is encountering issues.

QES tasks cannot complete because our provider is encountering issues.

Jun <var data-var='date'>22</var>, <var data-var='time'>17:56</var> UTC Resolved - We’re seeing recovery across our internal metrics, and processing has now returned to full capacity. At this time, the issue appears to be resolved. Our current leading hypothesis is resource contention on a shared Amazon RDS instance that several services depend on, potentially related to a VACUUM operation running alongside a long-running job deleting a large volume of accumulated historical data. We have not yet confirmed the root cause and will continue investigating as follow-up, but service has been restored for now. We'll be following up with a public post-mortem. Jun <var data-var='date'>22</var>, <var data-var='time'>17:44</var> UTC Monitoring - We're currently monitoring the EU cluster after an Amazon RDS issue.

Severe service disruption across the platform in EU

Jun <var data-var='date'>17</var>, <var data-var='time'>11:00</var> UTC Resolved - Incident was solved.

Sporadic PDF generation issue

Jun <var data-var='date'>17</var>, <var data-var='time'>09:23</var> UTC Resolved - The QES tasks TaT is back to normal. Jun <var data-var='date'>17</var>, <var data-var='time'>08:48</var> UTC Monitoring - Our provider was able to fix the issue and we don't get any error when contacting them. The TaT of QES tasks is back to normal. Monitoring the situation to make sure the errors don't come back. Jun <var data-var='date'>17</var>, <var data-var='time'>08:17</var> UTC Update - We're working closely with our provider to find the reason of the remaining errors. The error rate contacting our provider is stable and the impact on the QES task TaT is under control. Jun <var data-var='date'>17</var>, <var data-var='time'>06:50</var> UTC Update - Our provider's error rate dropped significantly. The TaT of QES tasks is now close to the usual value. Jun <var data-var='date'>17</var>, <var data-var='time'>06:19</var> UTC Update - Our provider is still working on fixing the issue. We're monitoring the impact and we're making sure to keep the TaT as low as possible, considering the situation. Jun <var data-var='date'>17</var>, <var data-var='time'>05:30</var> UTC Update - Our provider is actively working on fixing the issue. Jun <var data-var='date'>17</var>, <var data-var='time'>04:59</var> UTC Identified - One of our providers is still experiencing issues, resulting in higher TaT for electronic signature tasks.

Electronic signature increased TaT

Jun <var data-var='date'>16</var>, <var data-var='time'>23:17</var> UTC Resolved - The service is operational again. Jun <var data-var='date'>16</var>, <var data-var='time'>22:52</var> UTC Update - Our provider is still not able to answer to all requests, resulting in longer-than-usual electronic signature task processing time. Jun <var data-var='date'>16</var>, <var data-var='time'>22:19</var> UTC Identified - Our electronic signature tasks are taking more time to complete because one of our providers is doing some maintenance.

Electronic signature high TaT

Jun <var data-var='date'> 6</var>, <var data-var='time'>15:49</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'> 6</var>, <var data-var='time'>15:05</var> UTC Monitoring - The issue has been fixed, and the reports are being processed correctly. We are still monitoring the service. Jun <var data-var='date'> 6</var>, <var data-var='time'>14:36</var> UTC Identified - We identified the issue and are implementing a fix. Jun <var data-var='date'> 6</var>, <var data-var='time'>14:08</var> UTC Investigating - Some Watchlist search search profiles are not working correctly the impacted reports are not being processed. We are investigation the root cause.

Partial outage for Watchlist reports

May <var data-var='date'>12</var>, <var data-var='time'>13:14</var> UTC Resolved - All reports are back to normal. May <var data-var='date'>12</var>, <var data-var='time'>12:55</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. All clear rates should be back to normal May <var data-var='date'>11</var>, <var data-var='time'>23:11</var> UTC Identified - We are facing issues with one of our providers and we see a slight decrease in clear rate for Identity Reports in UK jurisdiction

Degraded performance for Identity Reports in UK jurisdiction

May <var data-var='date'> 9</var>, <var data-var='time'>12:03</var> UTC Resolved - The service is operational, no error where detected after the fix. May <var data-var='date'> 9</var>, <var data-var='time'>11:32</var> UTC Monitoring - Our provider fixed the issue. QES tasks are completing now. Tasks started during the incident were resumed. May <var data-var='date'> 9</var>, <var data-var='time'>11:27</var> UTC Update - The provider is working on a fix. May <var data-var='date'> 9</var>, <var data-var='time'>11:27</var> UTC Identified - We cannot complete QES tasks because our electronic signature provider is encountering issues.

May <var data-var='date'> 5</var>, <var data-var='time'>08:39</var> UTC Resolved - QES is operational again. May <var data-var='date'> 5</var>, <var data-var='time'>08:04</var> UTC Monitoring - The issue has been fixed. We're monitoring and resuming blocked QES tasks, if they can be. May <var data-var='date'> 5</var>, <var data-var='time'>07:35</var> UTC Update - Our provider is fixing the issue. QES tasks still cannot complete. May <var data-var='date'> 5</var>, <var data-var='time'>06:56</var> UTC Identified - Our provider is experiencing issues. All QES tasks cannot complete for now. May <var data-var='date'> 5</var>, <var data-var='time'>06:52</var> UTC Investigating - We noticed an issue with our QES tasks, we're investigating.

QES tasks outage

Apr <var data-var='date'>21</var>, <var data-var='time'>17:12</var> UTC Resolved - QES is operational again. Apr <var data-var='date'>21</var>, <var data-var='time'>17:02</var> UTC Update - Our provider is slowly recovering, we expect some QES tasks to complete. Apr <var data-var='date'>21</var>, <var data-var='time'>16:12</var> UTC Update - Our provider continues experiencing issues and is working on a fix. Apr <var data-var='date'>21</var>, <var data-var='time'>15:14</var> UTC Update - Our provider is still experiencing issues. All QES tasks cannot complete for now. Apr <var data-var='date'>21</var>, <var data-var='time'>14:28</var> UTC Identified - We identified an issue with one of our provider and some QES tasks may encounter some problems.

QES tasks degraded performance

Feb <var data-var='date'> 3</var>, <var data-var='time'>13:16</var> UTC Resolved - We can confirm the error stopped and services are stable and operational. Feb <var data-var='date'> 3</var>, <var data-var='time'>12:46</var> UTC Monitoring - The provider fixed the issue. We can confirm our QES tasks are now processed correctly. We'll monitor the situation to make sure the error is indeed fixed. Feb <var data-var='date'> 3</var>, <var data-var='time'>12:16</var> UTC Update - The provider is still working on a fix. QES verification tasks may start to time out, depending on their configuration, as we passed 90 minutes of downtime. Feb <var data-var='date'> 3</var>, <var data-var='time'>11:44</var> UTC Update - The provider is still working on a fix. Feb <var data-var='date'> 3</var>, <var data-var='time'>11:13</var> UTC Update - The provider is working on a fix, QES is still suffering a partial outage. Feb <var data-var='date'> 3</var>, <var data-var='time'>10:42</var> UTC Identified - One of our provider is experiencing issues. QES capture tasks will fail and QES verification tasks will have an increase in TaT.

QES tasks partial outage

Feb <var data-var='date'> 2</var>, <var data-var='time'>17:24</var> UTC Resolved - QES is operational again. Feb <var data-var='date'> 2</var>, <var data-var='time'>17:18</var> UTC Update - We're now seeing the error rate decrease. Most of the QES tasks should complete. Feb <var data-var='date'> 2</var>, <var data-var='time'>16:46</var> UTC Monitoring - Our provider is slowly recovering, we expect some QES tasks to complete. Feb <var data-var='date'> 2</var>, <var data-var='time'>16:15</var> UTC Update - Our provider is still experiencing issues. All QES tasks cannot complete for now. Feb <var data-var='date'> 2</var>, <var data-var='time'>15:36</var> UTC Identified - We identified an issue with one of our provider and some QES tasks may encounter some problems.

Jan <var data-var='date'>28</var>, <var data-var='time'>15:22</var> UTC Resolved - Incident is fully resolved, manual processing is now working normally. Manual reports will keep having an increased turn around time for a few more hours while it works through the task backlog. Jan <var data-var='date'>28</var>, <var data-var='time'>11:58</var> UTC Monitoring - Issue found and fixed. Increased turn around time for manual reports. Estimated time to live manual processing is 4h. Jan <var data-var='date'>28</var>, <var data-var='time'>11:16</var> UTC Investigating - We are currently investigating this issue.

Service Degradation - Manual Tasks

Jan <var data-var='date'>26</var>, <var data-var='time'>12:00</var> UTC Resolved - Between 12:15 UTC and 12:40 UTC there was disruption to document report processing in EU. Autofill requests also saw a disruption between 12:16 UTC and 12:27 UTC. A postmortem will follow.

Document report processing disrupted in EU

Jan <var data-var='date'>21</var>, <var data-var='time'>12:34</var> UTC Resolved - This issue is now resolved: Between 10:30 and 12:00 UTC, document reports could not be processed and autofill requests were failing in the US region. Starting from 12:00, live traffic was unaffected. From 12:00 to 12:30 UTC, pending reports were processed. We take a lot of pride in running a robust, reliable service and we're working hard to make sure this does not happen again. A detailed postmortem will follow once we've concluded our investigation. Jan <var data-var='date'>21</var>, <var data-var='time'>12:07</var> UTC Monitoring - We have deployed a fix. New document reports and autofill requests are being processed normally. We will rerun the earlier impacted reports. Jan <var data-var='date'>21</var>, <var data-var='time'>11:49</var> UTC Identified - We have identified the issue as a configuration issue between services, and are working on a fix. Jan <var data-var='date'>21</var>, <var data-var='time'>11:05</var> UTC Investigating - We have a high error rate in document report processing and autofill. Our team is investigating

Document Reports not being processed in US

Jan <var data-var='date'>21</var>, <var data-var='time'>10:30</var> UTC Resolved - Between 10:30 UTC and 10:47 UTC there was disruption to document report processing in EU, as a side effect of reports delayed from the earlier incident being re-processed. Further details will follow in a postmortem.

Jan <var data-var='date'>21</var>, <var data-var='time'>10:00</var> UTC Resolved - Between 10:05 UTC and 10:18 UTC we experienced delays for a significant part of document reports, as well as errors on our Autofill product. This was due to the release of a faulty component. The release was rolled back to resolve the situation. The delayed reports backlog is being processed.

Delays in Document report processing (all Regions)

Jan <var data-var='date'>14</var>, <var data-var='time'>01:30</var> UTC Resolved - Video facial similarity tasks are taking longer than expected to complete

High Processing times for Video Facial Similarity Checks

Jan <var data-var='date'> 7</var>, <var data-var='time'>16:21</var> UTC Resolved - This incident has been resolved. We apologise for the service disruption. We'll share a public post-mortem later. Jan <var data-var='date'> 7</var>, <var data-var='time'>16:00</var> UTC Update - We continue to monitor the recovery and work on processing reports delayed during the incident. Jan <var data-var='date'> 7</var>, <var data-var='time'>15:57</var> UTC Update - Processing has recovered across the platform. Jan <var data-var='date'> 7</var>, <var data-var='time'>15:45</var> UTC Update - Multiple products are stable, but still see some degraded performance. Jan <var data-var='date'> 7</var>, <var data-var='time'>15:43</var> UTC Update - Further recovery steps have been taken to stabilize the platform. Jan <var data-var='date'> 7</var>, <var data-var='time'>15:29</var> UTC Update - We are recovering, but still have high error rates. We continue to investigate residual issues. Jan <var data-var='date'> 7</var>, <var data-var='time'>14:28</var> UTC Monitoring - A fix has been implemented and we are monitoring. Jan <var data-var='date'> 7</var>, <var data-var='time'>13:39</var> UTC Identified - We have identified the issue and we are working on a fix Jan <var data-var='date'> 7</var>, <var data-var='time'>13:13</var> UTC Investigating - We are currently investigating an issue that is causing service disruption across multiple products.

Dec <var data-var='date'>19</var>, <var data-var='time'>11:00</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'>19</var>, <var data-var='time'>10:52</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Dec <var data-var='date'>19</var>, <var data-var='time'>10:39</var> UTC Update - We are continuing to work on a fix for this issue. Dec <var data-var='date'>19</var>, <var data-var='time'>10:35</var> UTC Identified - The issue has been identified and we are proceeding to the fix. We will provide an update in 15 minutes.

Errors when processing checks in Canada region

Dec <var data-var='date'>18</var>, <var data-var='time'>12:31</var> UTC Resolved - The incident was resolved Dec <var data-var='date'>18</var>, <var data-var='time'>11:49</var> UTC Update - Performance of the Evidence File service has been fully restored. However, the Qualified Electronic Signature (QES) service continues to experience significant degradation due to ongoing issues on the QTSP provider’s side. We are actively monitoring the situation and will share further updates as soon as possible. Dec <var data-var='date'>18</var>, <var data-var='time'>11:26</var> UTC Monitoring - Our QTSP provider is currently experiencing technical issues that are impacting the performance of our Qualified Electronic Signature (QES) and Evidence File services. We are actively monitoring the situation and will provide updates as soon as more information becomes available.

Degraded performance of QES and Evidence File products

Dec <var data-var='date'>11</var>, <var data-var='time'>23:00</var> UTC Resolved - A recently enabled feature in our services caused an unexpected increase in source integrity breakdown rejections on Facial Similarity reports, due to payload integrity verification (used to flag network injection attacks). This anomaly resulted in 5K reports being incorrectly flagged as “consider” for customers using the Standard and Photo Fully Auto variants of Facial Similarity reports, and only for clients that leverage the Device Intelligence report.

Increase in Payload Integrity Flagging Rate

Dec <var data-var='date'> 9</var>, <var data-var='time'>13:46</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'> 9</var>, <var data-var='time'>13:33</var> UTC Identified - We accumulated a large backlog of messages to update OpenSearch and we are currently processing it. The backlog should be cleared in the next hours.

Is Onfido down?

Onfido status is UP

Lastest published events

Delays in processing reports in the US cluster

Identity Enhanced issues

QES tasks cannot complete because our provider is encountering issues.

Onfido Services, Components and Regions

Recent Onfido events

Delays in processing reports in the US cluster

Identity Enhanced issues

QES tasks cannot complete because our provider is encountering issues.

Severe service disruption across the platform in EU

Sporadic PDF generation issue

Electronic signature increased TaT