Apr <var data-var='date'>23</var>, <var data-var='time'>14:02</var> UTC Resolved - All cloud services are fully operational and stable. We apologize for the inconvenience caused by the outage. Apr <var data-var='date'>22</var>, <var data-var='time'>09:13</var> UTC Update - All cloud services are fully operational and stable. We have released PIX4Dcatch iOS version 2.23, which contains a permanent fix for the feature outages experienced in version 2.22.0. Action Required: to restore full functionality, please visit the App Store and update PIX4Dcatch to version 2.23. We appreciate your patience while we worked to resolve this. Apr <var data-var='date'>21</var>, <var data-var='time'>15:19</var> UTC Monitoring - We have restored most of the cloud services, and we are monitoring their status. Engineers are still working on a fix for Catch iOS 2.22.0. Some Catch features are disabled in this version. Apr <var data-var='date'>21</var>, <var data-var='time'>12:40</var> UTC Update - We have identified the root cause of the incident. Apr <var data-var='date'>21</var>, <var data-var='time'>10:27</var> UTC Update - We are continuing to investigate the issue. Apr <var data-var='date'>21</var>, <var data-var='time'>07:35</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently investigating the issue.

Elevated API Errors

Feb <var data-var='date'>28</var>, <var data-var='time'>12:01</var> UTC Completed - The scheduled maintenance has been completed. Feb <var data-var='date'>28</var>, <var data-var='time'>12:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>26</var>, <var data-var='time'>07:27</var> UTC Scheduled - As part of our ongoing efforts to enhance our development infrastructure, Pix4D is retiring our PIX4Dmapper legacy pipeline from cloud products. This transition aims to provide improved performance, security, and scalability. See support article: https://support.pix4d.com/hc/upcoming-decommissioning-of-pix4dmapper-legacy-processing-pipeline

Decommissioning Notice: Pix4Dmapper cloud pipeline

Feb <var data-var='date'>13</var>, <var data-var='time'>12:34</var> UTC Resolved - We have restored the annotations that were deleted during the time of the incident. We recommend refreshing the affected datasets. Feb <var data-var='date'>13</var>, <var data-var='time'>09:45</var> UTC Update - We have fixed the visual glitches of all affected datasets. We are proceeding with restoring the annotations of the affected datasets. Feb <var data-var='date'>12</var>, <var data-var='time'>15:38</var> UTC Update - We have identified a solution to restore deleted annotations and fix visual glitches. We are currently running tests. Feb <var data-var='date'>12</var>, <var data-var='time'>09:02</var> UTC Update - We are investigating solutions to restore the deleted annotations. We will communicate with the affected organizations. Feb <var data-var='date'>11</var>, <var data-var='time'>16:11</var> UTC Monitoring - Annotations were unexpectedly deleted for datasets where new annotations were created during the time of the incident. A fix has been implemented, so no more annotations will be unexpectedly deleted. Feb <var data-var='date'>11</var>, <var data-var='time'>08:28</var> UTC Investigating - We are currently investigating the issue.

Cloud UI - annotations unexpectedly deleted

Jan <var data-var='date'>22</var>, <var data-var='time'>09:23</var> UTC Resolved - All services are restored correctly with the revert of the change. We do not notice any further problem. Sorry for the inconvenience caused by the API outage. Jan <var data-var='date'>22</var>, <var data-var='time'>09:14</var> UTC Update - The services are back to normal. We are monitoring and validating the traffic. Jan <var data-var='date'>22</var>, <var data-var='time'>09:07</var> UTC Monitoring - The services are back online but we are investigating an issue on the authentication. Jan <var data-var='date'>22</var>, <var data-var='time'>09:01</var> UTC Update - We are working on restoring the cloud API and site. Jan <var data-var='date'>22</var>, <var data-var='time'>08:59</var> UTC Identified - The issue has been identified and a fix is being implemented.

Cloud API and site down

Dec <var data-var='date'> 7</var>, <var data-var='time'>08:30</var> UTC Completed - The maintenance is completed. Dec <var data-var='date'> 7</var>, <var data-var='time'>08:11</var> UTC Verifying - All databases have been upgraded smoothly. We are monitoring the services. Dec <var data-var='date'> 7</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>19</var>, <var data-var='time'>15:32</var> UTC Scheduled - In our commitment to improving service reliability, performance and security, we will perform an upgrade of our main databases. The operation will require a very short switch-over to the new clusters and we expect to see API failures during less than 1 minute within the maintenance window. To minimize the impact of such operation we are planning it during a low traffic time, outside of week working days.

Databases upgrade

Nov <var data-var='date'>25</var>, <var data-var='time'>11:02</var> UTC Update - The maintenance is finished. We will keep observing the traffic all day and perform adjustment when needed. Nov <var data-var='date'>25</var>, <var data-var='time'>11:00</var> UTC Completed - The scheduled maintenance has been completed. Nov <var data-var='date'>25</var>, <var data-var='time'>10:47</var> UTC Verifying - We partially applied the maintenance and are currently observing the traffic. Nov <var data-var='date'>25</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>11</var>, <var data-var='time'>15:51</var> UTC Scheduled - In our commitment to improving service reliability and performance, we are moving our API traffic to a new infrastructure. This upgrade aims to provide improved reliability, performance, and scalability for the Cloud API. During that time, we do not expect any interruption in the Cloud API. The traffic will be moved transparently.

API Infrastructure Upgrade

Nov <var data-var='date'>10</var>, <var data-var='time'>11:04</var> UTC Resolved - We are closing the incident as resolved. Nov <var data-var='date'>10</var>, <var data-var='time'>10:31</var> UTC Monitoring - The fix has been applied and the load decreased immediately. This change was a modification we did not rollback entirely on Friday. We are monitoring the situation. Nov <var data-var='date'>10</var>, <var data-var='time'>09:54</var> UTC Identified - We found the issue and are working on deploying a patch. Nov <var data-var='date'>10</var>, <var data-var='time'>08:17</var> UTC Investigating - We are currently observing a high load on our database, we are investigating the cause.

High database load

Nov <var data-var='date'> 7</var>, <var data-var='time'>16:11</var> UTC Resolved - The load is resolved now. The incident is closed now. Nov <var data-var='date'> 7</var>, <var data-var='time'>15:25</var> UTC Update - The mitigation is fully in place. We are continuing to monitor the situation to ensure the database load remains stable. Nov <var data-var='date'> 7</var>, <var data-var='time'>15:19</var> UTC Update - The mitigation is not yet fully deployed. The load is getting better but still not at its nominal level. Nov <var data-var='date'> 7</var>, <var data-var='time'>14:41</var> UTC Monitoring - The mitigation is being deployed. We are monitoring the load to ensure it will come back to its nominal level. Nov <var data-var='date'> 7</var>, <var data-var='time'>14:19</var> UTC Identified - The issue has been identified and we working to mitigate it as it slows down the API response time. Nov <var data-var='date'> 7</var>, <var data-var='time'>13:52</var> UTC Investigating - We are currently observing a high load on our database, we are investigating the cause.

Oct <var data-var='date'>20</var>, <var data-var='time'>10:13</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>07:00</var> UTC Identified - Our cloud provider (AWS) is experiencing technical issues on multiple services we rely on, affecting the API and the processing infrastructure. Processing might be delayed or failing as a result. For more details please check AWS incident page: https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_BA540_514A652BE1A

AWS Outage - Elevated errors on our backend infrastructure

Oct <var data-var='date'>13</var>, <var data-var='time'>11:28</var> UTC Resolved - The issue has been identified and resolved. Oct <var data-var='date'>13</var>, <var data-var='time'>10:54</var> UTC Update - We are continuing to work on a fix for this issue. Oct <var data-var='date'>13</var>, <var data-var='time'>09:54</var> UTC Identified - Issue with interacting with the datasets from the Pix4Dcloud UI and Pix4Dcatch.

Pix4Dcloud and Pix4Dcatch outage

Sep <var data-var='date'>11</var>, <var data-var='time'>17:21</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'>11</var>, <var data-var='time'>17:07</var> UTC Update - We are continuing to monitor for any further issues. Sep <var data-var='date'>11</var>, <var data-var='time'>17:03</var> UTC Monitoring - A fix is getting deployed. We will test the fix in production as soon as it is ready. Sep <var data-var='date'>11</var>, <var data-var='time'>13:45</var> UTC Identified - API engine customers cannot process.

Engine API outage

Feb <var data-var='date'> 3</var>, <var data-var='time'>12:12</var> UTC Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 3</var>, <var data-var='time'>12:05</var> UTC Verifying - Verification is currently underway for the maintenance items. Feb <var data-var='date'> 3</var>, <var data-var='time'>12:02</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>27</var>, <var data-var='time'>14:41</var> UTC Scheduled - We will change our web security configuration to only allow embedding projects pages (/project/embeds/) within iframes to better secure the access to our cloud services and pages. This change will not impact the accessibility to our services. It's an informative maintenance for a security configuration change. In case of doubt, please reach out to our support.

Security - Configuration update to disallow iframes

Nov <var data-var='date'>21</var>, <var data-var='time'>10:14</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'>21</var>, <var data-var='time'>10:04</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Nov <var data-var='date'>21</var>, <var data-var='time'>09:46</var> UTC Identified - The issue has been identified and a fix is being implemented. Nov <var data-var='date'>21</var>, <var data-var='time'>09:32</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently investigating the issue.

Nov <var data-var='date'>21</var>, <var data-var='time'>09:13</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'>21</var>, <var data-var='time'>09:10</var> UTC Update - We are continuing to monitor for any further issues. Nov <var data-var='date'>21</var>, <var data-var='time'>09:04</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Nov <var data-var='date'>21</var>, <var data-var='time'>08:57</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently investigating the issue.

Oct <var data-var='date'> 1</var>, <var data-var='time'>13:07</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'> 1</var>, <var data-var='time'>12:53</var> UTC Investigating - We are currently investigating an issue of elevated error rates in PIX4Dcloud

Cloud elevated error rates

Jun <var data-var='date'>12</var>, <var data-var='time'>15:31</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>12:33</var> UTC Monitoring - The service is back to normal after performing an operation to restore the service in a healthy state. We are monitoring the situation. Jun <var data-var='date'>12</var>, <var data-var='time'>11:50</var> UTC Investigating - We're experiencing an elevated level of API errors on the license check endpoint and are currently investigating the issue.

Elevated API Errors on License check

Feb <var data-var='date'>19</var>, <var data-var='time'>11:01</var> UTC Resolved - The incident is resolved. Feb <var data-var='date'>19</var>, <var data-var='time'>09:05</var> UTC Monitoring - The error rate has decreased back to zero and we are monitoring the situation. Feb <var data-var='date'>19</var>, <var data-var='time'>09:00</var> UTC Identified - The issue has been identified and we are seeing the error rate decreasing. Feb <var data-var='date'>19</var>, <var data-var='time'>08:50</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently investigating the issue.

Jan <var data-var='date'>23</var>, <var data-var='time'>11:05</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>23</var>, <var data-var='time'>10:39</var> UTC Monitoring - We have identified the cause and restored the DNS configuration. We are closely monitoring the resolution. We will restart projects that failed processing due to the incident. Jan <var data-var='date'>23</var>, <var data-var='time'>09:53</var> UTC Investigating - We are currently investigating DNS resolution failures for some customers connecting to pix4d.com

DNS resolution failure

Dec <var data-var='date'>15</var>, <var data-var='time'>14:15</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'>15</var>, <var data-var='time'>11:27</var> UTC Update - The changes have been reverted. We are continuing to monitor the situation to ensure DNS resolution across different world servers is working correctly. Dec <var data-var='date'>15</var>, <var data-var='time'>09:57</var> UTC Monitoring - A DNS change affected the DNS resolution of all our domains. We reverted the change and are monitoring DNS propagation. The changes were applied this morning at 9:31 UTC which resulted in an invalid DNS path for naming resolution of our domains.

Dec <var data-var='date'> 4</var>, <var data-var='time'>21:49</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'> 4</var>, <var data-var='time'>21:39</var> UTC Monitoring - The remediation is active and the service is responding again as normal. We are monitoring the traffic / error rate. Dec <var data-var='date'> 4</var>, <var data-var='time'>21:29</var> UTC Identified - The issue has been identified and a fix is being implemented. Dec <var data-var='date'> 4</var>, <var data-var='time'>21:15</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently investigating the issue.

Nov <var data-var='date'>30</var>, <var data-var='time'>15:04</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'>30</var>, <var data-var='time'>14:56</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Nov <var data-var='date'>30</var>, <var data-var='time'>14:51</var> UTC Identified - The issue has been identified and a fix is being implemented. Nov <var data-var='date'>30</var>, <var data-var='time'>14:34</var> UTC Update - We are continuing to investigate this issue. Nov <var data-var='date'>30</var>, <var data-var='time'>14:13</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently investigating the issue.

Sep <var data-var='date'>26</var>, <var data-var='time'>11:20</var> UTC Resolved - The incident is solved. The errors were temporary failure events during a deployment. Sep <var data-var='date'>26</var>, <var data-var='time'>11:08</var> UTC Identified - The issue causing the login failure has been identified, we are working to correct it. Sep <var data-var='date'>26</var>, <var data-var='time'>11:03</var> UTC Investigating - We are observing some errors for some user profiles on the log in action.

Few users failing to log in

Sep <var data-var='date'>21</var>, <var data-var='time'>08:37</var> UTC Resolved - The incident is resolved. All point cloud conversions have be re-triggered with the fix in place and are either in progress or already done. Sep <var data-var='date'>21</var>, <var data-var='time'>06:43</var> UTC Monitoring - A fix has been implemented and we will reprocess all the point clouds for the affected projects. Sep <var data-var='date'>21</var>, <var data-var='time'>06:37</var> UTC Identified - The issue has been identified and a fix is being implemented. Sep <var data-var='date'>21</var>, <var data-var='time'>06:35</var> UTC Update - We are continuing to investigate this issue. Sep <var data-var='date'>21</var>, <var data-var='time'>06:05</var> UTC Investigating - We observe an issue with the processing of point cloud for online visualization, and we are investigating.

Issue in processing of point cloud

Jul <var data-var='date'>17</var>, <var data-var='time'>16:16</var> UTC Resolved - This incident has been resolved. The Cloud UI and API are currently stable, we're working on hardening the system against similar issues. Jul <var data-var='date'>17</var>, <var data-var='time'>15:33</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>17</var>, <var data-var='time'>15:16</var> UTC Investigating - We are currently investigating this issue.

Cloud UI and API degraded performance

Jun <var data-var='date'>15</var>, <var data-var='time'>11:40</var> UTC Resolved - 2023-06-15 13:40 CEST - Investigating We are experiencing significant delays with queued tasks. The following tasks are affected: - Project processing - Project tasks - Invoice generation - Mail delivery 2023-06-15 14:45 CEST - Monitoring The queues are processing again as expected. The delays concern the tasks queued today between 8:15 and 14:40 CEST 2023-06-21 - 16:00 CEST - Resolved All the delayed tasks from June 15th have been recovered and re-triggered. No data was lost, but some emails will contain old or inconsistent dates because of the delay to reschedule them. This incident is now resolved.

Is Pix4D down?

Pix4D status is UP

Lastest published events

Elevated API Errors

Decommissioning Notice: Pix4Dmapper cloud pipeline

Cloud UI - annotations unexpectedly deleted

Pix4D Services, Components and Regions

Recent Pix4D events

Elevated API Errors

Decommissioning Notice: Pix4Dmapper cloud pipeline

Cloud UI - annotations unexpectedly deleted

Cloud API and site down

Databases upgrade

API Infrastructure Upgrade