Jul <var data-var='date'>24</var>, <var data-var='time'>13:02</var> CDT Resolved - This incident has been resolved. Jul <var data-var='date'>23</var>, <var data-var='time'>17:01</var> CDT Monitoring - We are continuing to make updates to enhance system performance and minimize latency on the history/audits tables in the US production environment. The current ETA for resumption of normal replication speed is approximately 10 hours. We will provide an update tomorrow morning to confirm status. Jul <var data-var='date'>23</var>, <var data-var='time'>12:31</var> CDT Update - Please note that the data replication delay is only for history/audits tables in the US production environment. Other tables are replicating normally. We are continuing to investigate possible optimizations and will provide an update within 4 hours. Jul <var data-var='date'>23</var>, <var data-var='time'>11:22</var> CDT Identified - We are experiencing data replication delays for history/audits tables in the US production environment. We have identified the cause and made enhancements to improve performance. The current ETA for resumption of normal replication speed is 18 - 24 hours, and we are continuing to work on further optimizations. We will provide an update within 4 hours.

Data Replication Delay - US Production

Jun <var data-var='date'>18</var>, <var data-var='time'>06:26</var> CDT Resolved - The delay in US UAT data replication has been resolved, and performance and returned to normal levels. Jun <var data-var='date'>17</var>, <var data-var='time'>21:21</var> CDT Update - We are continuing to progress on the fix for the issue and performance is improving. We will provide another update by 9am central. Jun <var data-var='date'>17</var>, <var data-var='time'>18:23</var> CDT Identified - We are experiencing a delay in US UAT data replication of approximately 2.5 hours. We are actively working on resolving the issue and will provide another update within 2 hours.

US UAT Data Replication Delay

Jun <var data-var='date'>10</var>, <var data-var='time'>16:57</var> CDT Resolved - This incident has been resolved and access restored. Please reach out to Snapsheet Support with any further questions or concerns. Jun <var data-var='date'>10</var>, <var data-var='time'>16:34</var> CDT Investigating - We have identified an issue within the UAT environment that is impacting user access. We are investigating the issue as a high priority and will provide an update within one hour.

UAT Environment Outage

May <var data-var='date'> 7</var>, <var data-var='time'>15:51</var> CDT Resolved - Data replication latency has returned to normal levels. We will review the incident and follow up with a post-mortem within the next business week. Please contact the Snapsheet support team if you experience any further issues. May <var data-var='date'> 7</var>, <var data-var='time'>14:10</var> CDT Identified - We are currently experiencing data replication latency in the production environment. We are aware of the issue and working to improve performance. We will provide another update within the next two hours.

Data Latency Delay

Apr <var data-var='date'>15</var>, <var data-var='time'>09:30</var> CDT Resolved - The data inconsistencies have been addressed, and the incident is resolved. Apr <var data-var='date'>14</var>, <var data-var='time'>17:29</var> CDT Update - We are continuing to address the data inconsistencies. We will provide another update by 8pm CDT. Apr <var data-var='date'>14</var>, <var data-var='time'>14:03</var> CDT Update - We are continuing to address the data inconsistencies. We will provide another update at 5pm CDT. Apr <var data-var='date'>14</var>, <var data-var='time'>09:38</var> CDT Identified - We have identified some remaining data inconsistencies from the replication delay experienced on April 13. We are working to address the issue with an ETR of end of business today. We will provide another update by 2pm CDT today.

Data Reconciliation

Apr <var data-var='date'>14</var>, <var data-var='time'>07:45</var> CDT Resolved - This incident has been resolved. Apr <var data-var='date'>13</var>, <var data-var='time'>16:40</var> CDT Monitoring - We have resolved the data replication latency. All new data is replicating, and we are working to confirm data integrity for the time during which the issue occurred. We will provide another update within two hours. Apr <var data-var='date'>13</var>, <var data-var='time'>15:02</var> CDT Investigating - We have identified a delay in data replication, and we are are working to address it as a high priority. We will provide an update by 5pm CDT.

Data Replication Delay

Feb <var data-var='date'>26</var>, <var data-var='time'>09:37</var> CST Resolved - Sendgrid has resolved the issue that is causing delayed email deliveries to some recipients with Microsoft associated inboxes, and pending emails have been delivered. Please contact Snapsheet Support with any further questions or concerns. Feb <var data-var='date'>25</var>, <var data-var='time'>15:59</var> CST Identified - We have identified the issue that is causing delayed email deliveries to some recipients with Microsoft associated inboxes. We are seeing some improved performance, but we are continuing to work with Sendgrid to fully resolve the issue. We will provide another update by 10am CST tomorrow. Feb <var data-var='date'>25</var>, <var data-var='time'>12:40</var> CST Update - We have engaged Sendgrid to remediate the issue that is causing the delays in email deliveries. The next update will be no later than 4pm CST. Feb <var data-var='date'>25</var>, <var data-var='time'>11:39</var> CST Investigating - We have observed an issue with Sendgrid that is causing some customers to experience degraded performance of Microsoft Outlook. This issue is resulting in pending deliveries of some emails from Snapsheet to the impacted Microsoft accounts.

Sendgrid Email Deferment Impacting Some Microsoft Office Users

Feb <var data-var='date'>23</var>, <var data-var='time'>18:13</var> CST Resolved - We have resolved the delay in search indexing and confirmed searches are performing normally. Feb <var data-var='date'>23</var>, <var data-var='time'>16:37</var> CST Investigating - There is currently a delay in search indexing that is causing a delay in claims or policies becoming searchable. We are aware of the issue and working to address it as a high priority.

Search Indexing Delay

Feb <var data-var='date'> 2</var>, <var data-var='time'>20:11</var> CST Resolved - This incident has been resolved, and the latest UAT release has been successfully deployed. Please contact Snapsheet Support if you experience and further issues. Feb <var data-var='date'> 2</var>, <var data-var='time'>18:56</var> CST Monitoring - We have successfully deployed the latest UAT release, and we are monitoring to confirm no further issues. Feb <var data-var='date'> 2</var>, <var data-var='time'>17:02</var> CST Identified - We have confirmed that users may experience intermittent issues accessing claims in UAT. We have identified that the issues are related to an interruption with the UAT deployment caused by a Github incident (https://www.githubstatus.com/). We will continue to monitor and work to complete the deployment as soon as possible.

Intermittent UAT Access Issues

Jan <var data-var='date'>28</var>, <var data-var='time'>10:35</var> CST Resolved - We have received confirmation from SendGrid that the increased Microsoft inbox provider blocks are due to an issue with Microsoft, and that the issue is not isolated to SendGrid’s environment nor specific to their sending configuration. While we are seeing improvement (reduction) of deferrals, Microsoft customers who are continuing to observe excessive deferrals and/or blocks should fill out the OLC Support request form at https://olcsupport.office.com/. Thank you for your patience and understanding regarding this issue. If you have any further questions or concerns, please contact Snapsheet Support for assistance. Jan <var data-var='date'>27</var>, <var data-var='time'>13:59</var> CST Update - We are seeing a some recovery in email deliverability following the disruption with our email delivery partner Sendgrid. While we are still awaiting official confirmation of a final resolution. We will continue to monitor the situation closely. Jan <var data-var='date'>23</var>, <var data-var='time'>08:12</var> CST Update - Sendgrid is continuing to investigate the issue. Updates from Sendgrid are available at https://status.sendgrid.com/. Jan <var data-var='date'>22</var>, <var data-var='time'>19:56</var> CST Update - Sendgrid is continuing to investigate the issue. Updates from Sendgrid are available at https://status.sendgrid.com/. Jan <var data-var='date'>22</var>, <var data-var='time'>17:34</var> CST Identified - Sendgrid is currently investigating an issue with Microsoft that is causing some customers to experience degraded performance of Microsoft Outlook. This issue is resulting in failed deliveries of some emails from Snapsheet to the impacted Microsoft accounts. Updates from Sendgrid are available at https://status.sendgrid.com/.

Sendgrid Email Blocks Impacting Some Microsoft Office Users

Dec <var data-var='date'> 1</var>, <var data-var='time'>11:34</var> CST Resolved - The incident is confirmed to be resolved. Dec <var data-var='date'> 1</var>, <var data-var='time'>10:38</var> CST Monitoring - We have deployed a fix and continuing to monitor to confirm that there are no ongoing issues. Dec <var data-var='date'> 1</var>, <var data-var='time'>10:11</var> CST Identified - We have identified the issue and fix. We are working to implement the fix and continuing to monitor. Dec <var data-var='date'> 1</var>, <var data-var='time'>09:22</var> CST Investigating - We are currently investigating an issue with API calls receiving errors. We are treating the issue as Urgent and will provide an update within one hour.

API Response Errors

Nov <var data-var='date'>18</var>, <var data-var='time'>10:19</var> CST Resolved - The Cloudflare outage has been updated to resolved, and we've confirmed Smartcomm letters are sending successfully. Nov <var data-var='date'>18</var>, <var data-var='time'>09:12</var> CST Update - We are continuing to monitor for any further issues. Nov <var data-var='date'>18</var>, <var data-var='time'>09:12</var> CST Monitoring - There is a Cloudflare outage that is impacting Smartcomms and causing letters to fail. Updates are provided by Cloudflare on their status page at https://www.cloudflarestatus.com/. We will continue to monitor and update our own status page accordingly.

Cloudflare Outage

Oct <var data-var='date'>30</var>, <var data-var='time'>12:53</var> CDT Resolved - This incident has been resolved. If you experience any further issues, please reach out to Snapsheet Support. Oct <var data-var='date'>30</var>, <var data-var='time'>11:11</var> CDT Monitoring - We are seeing improved performance and continuing to monitor. We will provide additional updates as they become available. Oct <var data-var='date'>30</var>, <var data-var='time'>10:53</var> CDT Investigating - We are seeing a delay in data replication and are investigating it as a high priority. We will provide updates as they become available.

Oct <var data-var='date'>20</var>, <var data-var='time'>21:31</var> CDT Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>21:31</var> CDT Update - The AWS outage has been confirmed to be resolved. Please reach out to Snapsheet Support with any questions or concerns. Oct <var data-var='date'>20</var>, <var data-var='time'>12:03</var> CDT Update - We are continuing to monitor the ongoing AWS outage. As a result of this incident, we are experiencing delays in data warehouse replication. We'll share updates as more information becomes available. Oct <var data-var='date'>20</var>, <var data-var='time'>09:59</var> CDT Investigating - We are aware of an ongoing outage within Amazon Web Services (AWS). Our team is actively monitoring for any potential impact to our systems and will provide updates as more information becomes available from AWS.

Monitoring Potential Impact from Ongoing AWS Outage

Aug <var data-var='date'>18</var>, <var data-var='time'>17:53</var> CDT Resolved - All of the pending transactions impacted by the issue have been transitioned, and the incident is considered Resolved. Please contact Snapsheet Support if you continue to experience any further issues. Aug <var data-var='date'>18</var>, <var data-var='time'>14:52</var> CDT Monitoring - OFAC has updated their certificate to resolve the error. We are currently transitioning the payments and will provide an update when the transitions are complete. Aug <var data-var='date'>18</var>, <var data-var='time'>12:55</var> CDT Update - We are continuing to monitor OFAC's progress and will provide updates here as we receive them. Aug <var data-var='date'>18</var>, <var data-var='time'>09:54</var> CDT Identified - OFAC is experiencing an outage with the OFAC API from trade.gov. They are experiencing a certificate error on their platform and currently looking into it. Payments created with the feature on will be put in Compliance Hold until a user approves them.

OFAC API

Feb <var data-var='date'> 8</var>, <var data-var='time'>22:30</var> CST Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 8</var>, <var data-var='time'>21:00</var> CST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'> 7</var>, <var data-var='time'>16:00</var> CST Scheduled - We will be performing maintenance on the Snapsheet Payments service database on February 8, 2025. The payments service will be temporarily inaccessible between 9pm and 10pm CT. Payment processing for payments issued that day will not be impacted.

Database Maintenance

Dec <var data-var='date'>18</var>, <var data-var='time'>18:51</var> CST Resolved - This incident has been resolved. If you experience any further issues, please contact Snapsheet support. Dec <var data-var='date'>18</var>, <var data-var='time'>15:14</var> CST Update - OFAC is continuing to work on a fix for this issue. Dec <var data-var='date'>18</var>, <var data-var='time'>15:14</var> CST Identified - OFAC is experiencing an outage with the OFAC API from trade.gov. They are experiencing high latency on their platform and currently looking into it. Payments created with the feature on will be put in Compliance Hold until a user approves them.

Sep <var data-var='date'>28</var>, <var data-var='time'>14:51</var> CDT Resolved - This issue has been resolved. Please contact Snapsheet Support if you experience any further issues. Sep <var data-var='date'>28</var>, <var data-var='time'>14:39</var> CDT Monitoring - The delays in replication have been resolved and we are continuing to monitor. Sep <var data-var='date'>28</var>, <var data-var='time'>13:55</var> CDT Update - We are seeing normal throughput for the data pipeline and expect the replication delays to be fully resolved shortly. Sep <var data-var='date'>28</var>, <var data-var='time'>13:08</var> CDT Update - We are continuing to work on addressing the replication delays and have engaged AWS support for Amazon's Data Migration Service where the delay is being experienced. Sep <var data-var='date'>28</var>, <var data-var='time'>11:30</var> CDT Identified - We are currently seeing delays in replication through the data warehouse. We actively working on addressing the delay and will provide another update as soon as possible.

Data Warehouse Delays

Sep <var data-var='date'>19</var>, <var data-var='time'>10:15</var> CDT Resolved - This incident has been resolved. Please contact Snapsheet Support if you experience any further issues. Sep <var data-var='date'>19</var>, <var data-var='time'>09:57</var> CDT Update - We are continuing to investigate the issue and will provide an update as soon as possible. Sep <var data-var='date'>19</var>, <var data-var='time'>08:51</var> CDT Investigating - “We are investigating an issue with the navigation bar in US UAT and will provide an update as soon as possible”

US UAT Navigation bar Issue

Sep <var data-var='date'>11</var>, <var data-var='time'>09:36</var> CDT Resolved - This incident has been resolved. Please contact Snapsheet Support if you experience any further issues. Sep <var data-var='date'>11</var>, <var data-var='time'>08:47</var> CDT Identified - “We are currently seeing delays in claims being indexed for search. We are investigating and will provide updates as they become available.”

Recent claims are unsearchable

Sep <var data-var='date'> 4</var>, <var data-var='time'>09:22</var> CDT Resolved - This incident has been resolved. Sep <var data-var='date'> 4</var>, <var data-var='time'>09:22</var> CDT Update - The processing delays have been resolved. We will continue to monitor performance and provide a post-mortem as soon as it becomes available. Sep <var data-var='date'> 4</var>, <var data-var='time'>08:55</var> CDT Investigating - We are currently experiencing processing delays with claim details showing up in the work queues.

Processing Delays

Jul <var data-var='date'>30</var>, <var data-var='time'>21:34</var> CDT Resolved - This incident has been resolved. We will continue to monitor performance and provide a post-mortem as soon as possible. Jul <var data-var='date'>30</var>, <var data-var='time'>21:00</var> CDT Monitoring - We are seeing improved response times and normal performance within Snapsheet Claims over the past half hour. We are still working with AWS until they confirm that the issue is resolved and to produce a full post-mortem for the issue. Jul <var data-var='date'>30</var>, <var data-var='time'>20:00</var> CDT Identified - We are remaining engaged with the AWS team as they continue to address the widespread AWS outage. They have identified the root cause and are actively working on multiple parallel paths to mitigate the issue. We will provide updates as they become available. Jul <var data-var='date'>30</var>, <var data-var='time'>18:56</var> CDT Update - We are continuing to investigate, but the issue appears to be caused by the AWS outage. We are working with the AWS team to get more information and will provide updates as they become available. Jul <var data-var='date'>30</var>, <var data-var='time'>17:57</var> CDT Update - We have identified that one of our underlying vendors, AWS, is reporting issues across multiple services: https://health.aws.amazon.com/health/status. We are continuing to investigate and will continue to provide updates as they become available. Jul <var data-var='date'>30</var>, <var data-var='time'>17:35</var> CDT Investigating - We are currently experiencing issues with accessing Snapsheet Claims. We are investigating urgently and will provide an update as soon as possible.

Snapsheet Claims Outage

Jul <var data-var='date'>11</var>, <var data-var='time'>23:00</var> CDT Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>11</var>, <var data-var='time'>21:00</var> CDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>11</var>, <var data-var='time'>15:00</var> CDT Scheduled - We hope this message finds you well. We are writing to inform you about a scheduled maintenance window for our database systems. This maintenance is necessary to ensure continued optimal performance and security of our services. Maintenance Details: Date: Thursday, July 11th Time: 9 PM to 11PM CDT Impact on Services: While we aim to minimize the interruption, there may be temporary delays in data warehouse replication and intermittent disruption to dashboard access during the maintenance window What You Need to Do: Avoid scheduling any critical tasks that depend on our database services during this period. We apologize for any inconvenience this may cause and appreciate your understanding and cooperation. If you have any questions or concerns, please feel free to reach out to our support team. Thank you for your continued support. Best regards,

Jun <var data-var='date'>17</var>, <var data-var='time'>10:02</var> CDT Resolved - This incident has been resolved. Jun <var data-var='date'>17</var>, <var data-var='time'>10:00</var> CDT Update - Incident Resolved Jun <var data-var='date'>17</var>, <var data-var='time'>09:34</var> CDT Investigating - We are currently experiencing an issue resulting in metrics dashboards not loading. We are investigating and will provide another update as soon as possible.

Metrics Showing Internal Error

Apr <var data-var='date'>28</var>, <var data-var='time'>04:19</var> CDT Resolved - The scheduled migrations have concluded. Apr <var data-var='date'>26</var>, <var data-var='time'>21:30</var> CDT Monitoring - On Saturday, April 27th, starting at approximately 2 am Central Standard Time, scheduled data migrations will take place, which may result in temporary delays in the data pipeline.

Is Snapsheet down?

Snapsheet status is UP

Lastest published events

Data Replication Delay - US Production

US UAT Data Replication Delay

UAT Environment Outage

Snapsheet Services, Components and Regions

Recent Snapsheet events

Data Replication Delay - US Production

US UAT Data Replication Delay

UAT Environment Outage

Data Latency Delay

Data Reconciliation

Data Replication Delay