Feb <var data-var='date'>26</var>, <var data-var='time'>00:32</var> UTC Resolved - The incident has been resolved. Issue determined to be caused due to a configuration change. Configuration change has been reverted.Feb <var data-var='date'>26</var>, <var data-var='time'>00:28</var> UTC Investigating - We are currently investigating the issue.

Census Web Outage

Jan <var data-var='date'>29</var>, <var data-var='time'>20:40</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'>29</var>, <var data-var='time'>19:30</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Jan <var data-var='date'>29</var>, <var data-var='time'>19:15</var> UTC Investigating - We are currently investigating this issue.

Degraded performance of Web & Sync Runs.

Jan <var data-var='date'>29</var>, <var data-var='time'>02:00</var> UTC Completed - The scheduled maintenance has been completed.Jan <var data-var='date'>29</var>, <var data-var='time'>01:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Jan <var data-var='date'>27</var>, <var data-var='time'>18:25</var> UTC Scheduled - We will be undergoing scheduled maintenance during this time.

Database Maintenance

Jan <var data-var='date'> 5</var>, <var data-var='time'>20:24</var> UTC Resolved - This incident has been resolved. Syncs have been running normally as of approximately 1:00pm EST, and the web has been running at normal latencies as of 2:00pm EST. The issue was caused by an indexing change on our side that impacted our ability to run syncs reliably. Recovery was delayed because we needed to rebuild the previous indexes. To avoid any risk of data loss, we chose a safer recovery approach that allowed the system to remain temporarily degraded while repairs were being completed. To restore syncs as quickly as possible, we prioritized sync execution and applied targeted fixes to improve performance. During this time, access to the UI was temporarily down.Jan <var data-var='date'> 5</var>, <var data-var='time'>19:05</var> UTC Monitoring - The web is now operating normally, along with Sync Runs. We are monitoring the system for overall health.Jan <var data-var='date'> 5</var>, <var data-var='time'>18:34</var> UTC Update - Syncs are now operational. Users may be experiencing issues loading Sync Configuration and Sync Run details on the web.Jan <var data-var='date'> 5</var>, <var data-var='time'>16:47</var> UTC Update - Syncs are currently not operational. Scheduled syncs should still be queuing up but they won't start running at this time. We will provide an update once workers come back up.Jan <var data-var='date'> 5</var>, <var data-var='time'>15:59</var> UTC Update - We are still seeing degraded performance and in some cases failing syncs. We are continuing efforts to mitigate as the system repairs.Jan <var data-var='date'> 5</var>, <var data-var='time'>15:27</var> UTC Update - We have identified and triaged the affected system and systems are coming back up now. Web is operational. We will see Sync delays as we scale back up.Jan <var data-var='date'> 5</var>, <var data-var='time'>15:17</var> UTC Identified - Census is currently having issues running Syncs and rendering the Website. Engineers have identified the issue and are remediating.

Census Full Outage

Dec <var data-var='date'>11</var>, <var data-var='time'>19:07</var> UTC Resolved - After investigation, a very small number of Sync Runs may have failed with an inability to connect to our Redis instance. The majority of syncs that ran between 1:19 pm and 1:28 pm EST will have experienced a few minutes of delay, and recovered automatically.Dec <var data-var='date'>11</var>, <var data-var='time'>18:39</var> UTC Monitoring - Census Web has recovered and we are analyzing Sync Runs for impact.Dec <var data-var='date'>11</var>, <var data-var='time'>18:32</var> UTC Identified - Census is experiencing an outage on web, and we are experiencing degraded sync run activity. We have identified the cause and have applied the fix. We are monitoring system recovery.

Census Web Outage and Sync Delays

Nov <var data-var='date'>12</var>, <var data-var='time'>20:01</var> UTC Resolved - This incident has been resolved.Nov <var data-var='date'>12</var>, <var data-var='time'>18:30</var> UTC Update - The error impacting some customer syncs has now been resolved and syncs are recovering. The syncs impacted were runs with no new data changes to process. This happened due to a recent update that removed the part of our system that correctly handled "no change" scenarios. As a result, syncs without updates failed instead of completing successfully. Syncs with actual data changes were unaffected. We've rolled back the change, and all syncs are now running normally.Nov <var data-var='date'>12</var>, <var data-var='time'>18:30</var> UTC Monitoring - The error impacting some customer syncs has now been resolved and syncs are recovering. The syncs impacted were runs with no new data changes to process. This happened due to a recent update that removed the part of our system that correctly handled "no change" scenarios. As a result, syncs without updates failed instead of completing successfully. Syncs with actual data changes were unaffected. We've rolled back the change, and all syncs are now running normally.Nov <var data-var='date'>12</var>, <var data-var='time'>17:41</var> UTC Identified - The issue has been identified and a fix is being implemented.

Subset of syncs are failing with, "Passed `nil` into T.must"

Oct <var data-var='date'>29</var>, <var data-var='time'>03:09</var> UTC Resolved - Delays have been resolved. Syncs are running normally again.Oct <var data-var='date'>29</var>, <var data-var='time'>02:53</var> UTC Identified - We’ve identified the cause of the minor delays and implemented a fix. Sync performance returning back to normal. We’re monitoring to ensure continued stability.Oct <var data-var='date'>29</var>, <var data-var='time'>02:13</var> UTC Investigating - We are currently investigating minor delays affecting some Census Sync job types in the US region. Most syncs are completing successfully, though some may take longer than usual.

Minor Delays in Census Syncs (US Region)

Oct <var data-var='date'>21</var>, <var data-var='time'>00:26</var> UTC Resolved - Our system has been operating normally for the last few hours, and the aforementioned downstream provider has marked their incident as resolved. We are marking the sync engine as operational. Please contact support if you believe you are continuing to see issues.Oct <var data-var='date'>20</var>, <var data-var='time'>20:51</var> UTC Update - Our systems show sustained recovery and remain stable. We will continue to monitor for any further issues.Oct <var data-var='date'>20</var>, <var data-var='time'>18:52</var> UTC Monitoring - We're now able to appropriately scale workers to meet demands. We are through our main backlogs (including mail sends), and Sync run latencies have improved substantially. We are still continuing to see transient errors throughout Sync runs, mainly dealing with issues hitting our Redis instances, but the frequency of these errors is continuing to decrease.Oct <var data-var='date'>20</var>, <var data-var='time'>17:54</var> UTC Update - We are seeing increases in Sync runtimes as we continue to experience issues adding new workers to our pool. We are continuing work to mitigate this issue.Oct <var data-var='date'>20</var>, <var data-var='time'>16:07</var> UTC Update - Non-sync background tasks are running normally. Email notifications are still delayed. We have detected syncs that are currently stuck in working state due to issues earlier in the day. We are working to unblock them.Oct <var data-var='date'>20</var>, <var data-var='time'>13:43</var> UTC Identified - In our US region, we are seeing increased latency in non-sync background tasks like metadata refreshes and connection tests due to issues scaling up worker load to match demand, as a result of the continued downstream provider outage. We have identified the issue and are now mitigating the impact.

Delays in Background Jobs

Oct <var data-var='date'>20</var>, <var data-var='time'>07:00</var> UTC Resolved - Census experienced major service degradation for running Syncs and on the web for users viewing Sync details for a period of about 3 hours, from 2:46 am EST until 5:30 am EST, due to a large-scale downstream provider outage. All systems are now operational.

Major Service Degradation

Sep <var data-var='date'>19</var>, <var data-var='time'>16:50</var> UTC Resolved - This incident has been resolved and page loads are now fully operational. During a system upgrade it was determined that our DNS resolver returned empty values for app.getcensus.com for a brief point - our team will continue investigating the root cause of why this occurred.Sep <var data-var='date'>19</var>, <var data-var='time'>16:41</var> UTC Investigating - We have detected intermittent page load failures on the Census UI due to DNS. Our team is investigating the root cause. Sync runs are unaffected.

Intermittent Page Load Fails

Jul <var data-var='date'>15</var>, <var data-var='time'>18:44</var> UTC Resolved - This incident has been resolved.Jul <var data-var='date'>15</var>, <var data-var='time'>18:31</var> UTC Monitoring - We have updated the our configurations for sending emails and we are currently verifying the fix.Jul <var data-var='date'>15</var>, <var data-var='time'>16:35</var> UTC Investigating - We have identified an incident that has caused Census to stop sending all emails. This includes emails related to sync alerts as well as password reset emails and verification emails for new account signups. We are working to remediate the issue. The Census UI and Sync Engine are not otherwise affected

Census unable to send any emails

Jul <var data-var='date'> 3</var>, <var data-var='time'>02:11</var> UTC Resolved - AWS has marked this as resolved.Jul <var data-var='date'> 3</var>, <var data-var='time'>01:58</var> UTC Monitoring - There are no known problems in production but we are monitoring this AWS incident presently as it has impacted some things in our staging environment. https://health.aws.amazon.com/health/status

AWS Incident Being Monitored - No Production Impact Yet

Jun <var data-var='date'>12</var>, <var data-var='time'>21:05</var> UTC Resolved - Census systems remain unaffected as GCP comes back online. We are now seeing the majority of our GCP hosted customers have successful syncs.Jun <var data-var='date'>12</var>, <var data-var='time'>20:19</var> UTC Update - Our monitoring systems continue to see a sustained error rate when communicating with Google BigQuery. We will monitor and report when the issue is resolved from Census' perspective. Google Cloud has provided additional information on their status page, located here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsWJun <var data-var='date'>12</var>, <var data-var='time'>18:37</var> UTC Update - Internal monitoring indicates that syncs to and from BigQuery started experiencing increased failure rates at around 18:05 UTC. The rate of failures is now decreasing but has not returned to usual levels.Jun <var data-var='date'>12</var>, <var data-var='time'>18:30</var> UTC Monitoring - We are aware of the current GCP outage and we are monitoring its impact on Census syncs to and from Google and Google Cloud Resources. Census does not run any critical components on GCP but we do use GCP resources, including Cloud Storage, to facilitate interaction with BigQuery and other Google Cloud resources.

GCP Outage May Affect Syncs to and from Google Resources

Jun <var data-var='date'> 6</var>, <var data-var='time'>18:18</var> UTC Resolved - This incident has been resolved.Jun <var data-var='date'> 6</var>, <var data-var='time'>17:52</var> UTC Monitoring - Some syncs started failing with the following error at 12:18pm (EST) "ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column sync_configurations.new_diff_timestamp_expression_id does not exist" This was due to a brief issue during a schema update. The problem has been resolved, and we are monitoring.

Sync Runs Failing with Undefined Column Error

Jun <var data-var='date'> 4</var>, <var data-var='time'>20:54</var> UTC Resolved - This incident has been resolved.Jun <var data-var='date'> 4</var>, <var data-var='time'>20:43</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Jun <var data-var='date'> 4</var>, <var data-var='time'>20:24</var> UTC Update - A fix for this issue is currently being deployed and will be in place in the next 15-30 minutesJun <var data-var='date'> 4</var>, <var data-var='time'>20:11</var> UTC Identified - The Census sync list UI is not loading for some customers. We have identified a faulty deployment and we are reverting it. The sync engine is not affected.

Sync list UI not loading

May <var data-var='date'>28</var>, <var data-var='time'>14:00</var> UTC Resolved - During a window of time from 9:58 am EST until 10:32 am EST, the Census web UI, and a portion of the Census Public Management API experienced an outage. Sync Runs were unaffected and operated normally.

Web Outage

Apr <var data-var='date'>21</var>, <var data-var='time'>19:33</var> UTC Resolved - A recent code change introduced a bug resulting in some syncs failing with the message "T.cast: Expected type T::Struct, got type TrueClass". This impacted sync runs where there were no incremental changes identified and the source connection utilizes the Advanced Sync Engine. We identified the problematic change, and have reverted. Syncs are now running successfully.

Some syncs failing with "T.cast: Expected type T::Struct, got type TrueClass"

Apr <var data-var='date'>16</var>, <var data-var='time'>23:54</var> UTC Resolved - This incident has been resolved.Apr <var data-var='date'>16</var>, <var data-var='time'>23:36</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Apr <var data-var='date'>16</var>, <var data-var='time'>23:31</var> UTC Identified - We've identified an issue that caused background jobs to be delayed. The issue does not impact running syncs.

Background job delays (Syncs not impacted)

Apr <var data-var='date'> 5</var>, <var data-var='time'>16:23</var> UTC Resolved - This incident has been resolved.Apr <var data-var='date'> 5</var>, <var data-var='time'>16:21</var> UTC Update - We are continuing to monitor for any further issues.Apr <var data-var='date'> 5</var>, <var data-var='time'>01:18</var> UTC Update - This issue is resolved.Apr <var data-var='date'> 4</var>, <var data-var='time'>23:00</var> UTC Monitoring - We experienced a performance issue from ~3:00-3:15pm Pacific that may have caused some syncs to fail with a message about "mismatch counts". We believe we have resolved the problem, but syncs that encountered the issue may not report an error until sometime after the fact, so errors may continue to trickle in. If you encounter this error, please attempt to rerun your sync. If a rerun does not succeed, please contact support.Apr <var data-var='date'> 4</var>, <var data-var='time'>22:40</var> UTC Investigating - We are currently investigating issue with some syncs failing with mismatch errors.

Sync might be failing with mismatch errors

Mar <var data-var='date'>26</var>, <var data-var='time'>05:43</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>26</var>, <var data-var='time'>04:46</var> UTC Update - The Census Store maintenance is now startingMar <var data-var='date'>26</var>, <var data-var='time'>04:04</var> UTC Update - This has not yet begun and is paused due to an unrelated incident.Mar <var data-var='date'>26</var>, <var data-var='time'>04:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>23</var>, <var data-var='time'>22:18</var> UTC Scheduled - We will be performing database maintenance for the Census Store Iceberg Catalog which will cause Census Store and technologies that depend on it (CSV uploads, SaaS datasets, and Entity Resolution) to be unavailable. The vast majority of Census syncs do not use Census Store and will not be affected by this maintenance. We do not expect any data loss during this maintenance, but reads and writes to Iceberg tables managed by Census Store will not be permitted and the Iceberg REST Catalog will be offline.

Census Store Maintenance

Mar <var data-var='date'>26</var>, <var data-var='time'>04:25</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>26</var>, <var data-var='time'>04:15</var> UTC Monitoring - We have reverted to the previous working version of the system and sync success rates have returned to normal levels. We are investigating why our integration tests were insufficient to prevent this issue.Mar <var data-var='date'>26</var>, <var data-var='time'>04:03</var> UTC Investigating - We have identified a bug that is causing "mirror" syncs to fail with an error in the SQL generated by Census; the error message varies by data warehouse, but relates to a missing column. We are rolling back the incorrect deployment. Note that this is unrelated to the scheduled maintenance for Census Store, which has not yet started and will be postponed until this issue is resolved.

Mirror Syncs Failing with SQL error

Mar <var data-var='date'>22</var>, <var data-var='time'>16:35</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>21</var>, <var data-var='time'>19:46</var> UTC Monitoring - We have applied a fix and functionality has been restored. We are continuing to monitor.Mar <var data-var='date'>21</var>, <var data-var='time'>17:57</var> UTC Identified - We have identified the cause and are working on a resolution.

New SaaS Dataset, CSV Dataset, and ER Dataset creation unavailable for some customers

Feb <var data-var='date'> 5</var>, <var data-var='time'>23:30</var> UTC Resolved - No errors have been seen for over 30 minutes. We are declaring the incident resolved.Feb <var data-var='date'> 5</var>, <var data-var='time'>22:56</var> UTC Monitoring - The revert has deployed. We are monitoring to ensure that the errors subside.Feb <var data-var='date'> 5</var>, <var data-var='time'>22:38</var> UTC Identified - A recent code change introduced a bug resulting in some syncs failing with the message "Missing corresponding dataset file for prefix: record_deletes.0". We have identified the problematic change, and a revert is in progress.

Some syncs failing with "Missing corresponding dataset file for prefix: record_deletes.0"

Jan <var data-var='date'>30</var>, <var data-var='time'>21:57</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'>30</var>, <var data-var='time'>21:44</var> UTC Investigating - Our monitoring has notified us that some syncs are performing more slowly than usual - we are investigating

Degraded sync performance

Jan <var data-var='date'>30</var>, <var data-var='time'>03:31</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'>30</var>, <var data-var='time'>03:24</var> UTC Monitoring - We have rolled back the service to a previous configuration and we are monitoring our system to ensure database / warehouse connectivity returns to normal levels.Jan <var data-var='date'>30</var>, <var data-var='time'>03:12</var> UTC Identified - We have identified the root cause and are applying a fix to the networking configuration.Jan <var data-var='date'>30</var>, <var data-var='time'>02:59</var> UTC Investigating - Due to a networking misconfiguration, requests originating from Census's database access service are originating from IPs that we have not previously advertised. If your database is configured to reject requests that are not from Census' IP addresses, syncs will start failing. We are investigating the root cause of the issue and it is our intent to restore service from the original IP addresses; we do not expect customers to need to reconfigure their databases at this time. More updates will be posted here.

Is Census down?

Census status is UP

Lastest published events

Census Web Outage

Degraded performance of Web & Sync Runs.

Database Maintenance

Census Services, Components and Regions

Recent Census events

Census Web Outage

Degraded performance of Web & Sync Runs.

Database Maintenance

Census Full Outage

Census Web Outage and Sync Delays

Subset of syncs are failing with, "Passed `nil` into T.must"

Get your FREE account now,
5 monitors included!

Is Census down?

Census status is UP

Lastest published events

Census Web Outage

Degraded performance of Web & Sync Runs.

Database Maintenance

Census Services, Components and Regions

Recent Census events

Census Web Outage

Degraded performance of Web & Sync Runs.

Database Maintenance

Census Full Outage

Census Web Outage and Sync Delays

Subset of syncs are failing with, "Passed `nil` into T.must"

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!