Aug <var data-var='date'>21</var>, <var data-var='time'>11:24</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'>21</var>, <var data-var='time'>11:00</var> PDT Monitoring - A fix has been deployed. Admins viewing Compensation Cycle Recommendations should refresh the page and see data successfully. We will continue monitoring for the time being. Aug <var data-var='date'>21</var>, <var data-var='time'>10:22</var> PDT Identified - Our team has identified the underlying root cause and and is implementing a fix to resolve the error admins see on the Compensation Cycle Recommendations page. Aug <var data-var='date'>21</var>, <var data-var='time'>09:30</var> PDT Investigating - Lattice is currently experiencing an incident on the admin Compensation Cycle Recommendations page. The Recommendations table is showing a "Something Went Wrong" error.

Admin Compensation Cycle Recommendations page showing "Something Went Wrong" error

Aug <var data-var='date'>20</var>, <var data-var='time'>12:30</var> PDT Resolved - Lattice experienced an issue with Lattice AI Agent and Evidence-Based Reviews generation between 12:30pm Pacific and 1:15pm Pacific. AI agent and Evidence-Based Reviews generation has been restored.

Lattice AI Agent and Evidence-Based Reviews generation errors

Aug <var data-var='date'>18</var>, <var data-var='time'>14:33</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'>18</var>, <var data-var='time'>14:32</var> PDT Monitoring - For approximately 30 minutes, email and Slack messages failed to send. A fix has been deployed, and new messages are now sending as expected. Messages sent during the affected window may be delayed. We are continuing to monitor the service.

Issues sending email and Slack notifications

Aug <var data-var='date'>14</var>, <var data-var='time'>10:52</var> PDT Resolved - From 10:52–11:36 AM PDT, some users may have experienced issues logging in, making changes in Lattice, or using API requests to create or update data. This issue has now been resolved.

Issues logging in and completing actions in Lattice

Aug <var data-var='date'>10</var>, <var data-var='time'>10:58</var> PDT Resolved - All delayed review cycle state updates and notifications have now been sent. Automated review schedules are operating on time. Aug <var data-var='date'>10</var>, <var data-var='time'>10:06</var> PDT Update - Lattice is currently processing all missed review cycle state updates and notifications. Review cycles will be moved to the proper intended status and reminder notifications will be sent for all scheduled events between Thursday, August 6th and today. Aug <var data-var='date'>10</var>, <var data-var='time'>08:20</var> PDT Identified - Our team has identified the underlying root cause and are implementing a fix. All missed review cycle state changes will be made and associated notifications will be delivered. Aug <var data-var='date'>10</var>, <var data-var='time'>06:39</var> PDT Investigating - Lattice is experiencing a disruption in automated review cycle schedules. All review cycles on an automated timeline have not moved their state forward and automated reminders have not sent since Thursday. Our team is actively working to identify the underlying root cause and implement a solution.

Automated Review Cycle Schedule Delays

Aug <var data-var='date'> 6</var>, <var data-var='time'>17:51</var> PDT Resolved - Slack notification settings have now been restored. Aug <var data-var='date'> 6</var>, <var data-var='time'>16:14</var> PDT Investigating - Some users who previously had Slack notifications active now have Slack notifications deactivated. These can be reactivated in Profile -> Manage Settings -> Notifications. Our engineering team is actively investigating.

Slack Settings Deactivated for Some Users

Aug <var data-var='date'> 5</var>, <var data-var='time'>13:14</var> PDT Resolved - This issue has been resolved. Aug <var data-var='date'> 5</var>, <var data-var='time'>11:07</var> PDT Identified - Our team has identified the underlying root cause and will be implementing a fix shortly. Aug <var data-var='date'> 4</var>, <var data-var='time'>20:37</var> PDT Investigating - After editing Default roles, the displayed counts for viewable and editable field-level permissions may be incorrect until the page is refreshed. As a workaround, refresh the page after saving. Our engineering team is actively investigating.

Issues saving Default role Field-Level Permissions

Jul <var data-var='date'>24</var>, <var data-var='time'>04:00</var> PDT Resolved - Cloudflare, an upstream provider, is currently experiencing an issue. This temporarily affected Lattice site performance and caused connection timeout errors for some users. Lattice performance has returned to normal. Our Engineering Team is continuing to monitor the situation.

Issue with upstream provider

Jun <var data-var='date'> 4</var>, <var data-var='time'>16:30</var> PDT Resolved - We are aware of an issue present over the weekend that meant that an error message appeared when attempting to end goals as complete or incomplete. Our Engineering Team have released a fix for this issue, and this has now been resolved.

Issue ending goals as complete or incomplete

May <var data-var='date'> 5</var>, <var data-var='time'>04:02</var> PDT Resolved - Our Engineering Team has investigated this issue, and we are seeing a restoration of performance across Lattice. We will be continuing to investigate the root cause behind the slow down. May <var data-var='date'> 5</var>, <var data-var='time'>02:45</var> PDT Investigating - Our Engineering Team is aware of degraded performance and long loading times for the login page, with some users experiencing long page load times for different areas in Lattice. We are investigating the issue.

Login and page loading issues

Apr <var data-var='date'>14</var>, <var data-var='time'>07:00</var> PDT Resolved - This incident has been resolved, users can now chat into Lattice and the FIN Bot will escalate to a live agent if needed. Apr <var data-var='date'>14</var>, <var data-var='time'>06:05</var> PDT Identified - The issue has been identified, users can chat in to Customer Care and they will go directly to a live agent for support. Apr <var data-var='date'>14</var>, <var data-var='time'>05:59</var> PDT Investigating - Our upstream provider, Intercom, is currently experiencing an issue that prevents our customers from reaching Customer Care via live chat. If you need immediate assistance, please email us at customercare@lattice.com.

Intercom Chat Bot down

Apr <var data-var='date'> 8</var>, <var data-var='time'>20:52</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'> 8</var>, <var data-var='time'>18:49</var> PDT Identified - The issue has been identified and a fix is being implemented.

Some admins may see errors on the Results of a Review Cycle.

Apr <var data-var='date'> 8</var>, <var data-var='time'>11:26</var> PDT Resolved - This issue has been resolved. Apr <var data-var='date'> 8</var>, <var data-var='time'>09:48</var> PDT Identified - Users may see a Something Went Wrong error when accessing Goals in Lattice. The issue has been identified, and a fix is being implemented.

Something Went Wrong errors on Goals pages

Mar <var data-var='date'>26</var>, <var data-var='time'>18:08</var> PDT Resolved - This incident has been resolved. Mar <var data-var='date'>26</var>, <var data-var='time'>15:49</var> PDT Identified - When viewing shared review packets, if a score has been calibrated, the manager rating reflects the calibrated rating. The issue has been identified and a fix is being implemented.

Shared review packets showing calibrated score

Mar <var data-var='date'>26</var>, <var data-var='time'>15:18</var> PDT Resolved - This incident has been resolved. Mar <var data-var='date'>26</var>, <var data-var='time'>13:48</var> PDT Identified - A solution has been identified and a fix is being implemented. Mar <var data-var='date'>26</var>, <var data-var='time'>13:35</var> PDT Investigating - We are experiencing an intermittent issue where the OpenAI model is not responding to requests, which is impacting the Lattice Agent. As a result, users may notice that the Agent occasionally does not respond to requests. We are currently investigating this issue.

AI Agent intermittently not responding to requests

Mar <var data-var='date'>25</var>, <var data-var='time'>19:26</var> PDT Resolved - This incident has been resolved. Mar <var data-var='date'>25</var>, <var data-var='time'>16:17</var> PDT Identified - Review packets are showing pre-calibrated scores, even when calibrated ratings are present. The issue has been identified and a fix is being implemented.

Review packets showing pre-calibrated scores

Mar <var data-var='date'>24</var>, <var data-var='time'>05:00</var> PDT Resolved - Users may have seen slowness or pages failing to load across the Lattice application between 8:00am and 8:05am ET. This incident has been mitigated and Lattice performance is back to normal. Our Engineering Team is investigating the root cause behind the slowdown.

Lattice Latency / Failure to load pages

Feb <var data-var='date'>25</var>, <var data-var='time'>19:30</var> PST Resolved - An issue with an upstream provider, AWS, caused an outage of the Lattice platform. We have confirmed the issue has been resolved from our upstream provider, and we are actively monitoring.

Feb <var data-var='date'>12</var>, <var data-var='time'>15:46</var> PST Resolved - This incident has been resolved. Feb <var data-var='date'>12</var>, <var data-var='time'>14:10</var> PST Identified - The issue has been identified and a fix is being implemented. Feb <var data-var='date'>12</var>, <var data-var='time'>14:00</var> PST Investigating - We are currently investigating an issue that prevents admins from exporting PowerPoint slides from Engagement survey results.

Error when exporting Engagement PowerPoint slides

Feb <var data-var='date'> 2</var>, <var data-var='time'>14:29</var> PST Resolved - Courier is reporting that a fix has been implemented and the issue has been resolved, and we are seeing the Lattice application redirect emails successfully. Our Engineering Team will continue to monitor the effects of this downtime on our services. All Lattice product notifications that were delayed as a result of this incident have now been delivered. Some notifications may have been delayed by up to one day. Feb <var data-var='date'> 2</var>, <var data-var='time'>13:34</var> PST Investigating - Our upstream provider, Courier, is currently experiencing an issue which is affecting our ability to successfully redirect email links. This has also caused a delay to all Lattice product notifications sent via email, MS Teams, or Slack. Our Engineering Team are aware of this issue, and are actively investigating the effects of this downtime on our services.

Feb <var data-var='date'> 1</var>, <var data-var='time'>07:59</var> PST Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 1</var>, <var data-var='time'>06:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>29</var>, <var data-var='time'>01:07</var> PST Scheduled - Scheduled maintenance will begin on Sunday, February 1st at 6:00AM Pacific Standard Time. Lattice will be unavailable for approximately 2 hours. Lattice will return to full functionality immediately after maintenance is completed.

Scheduled Maintenance

Jan <var data-var='date'> 9</var>, <var data-var='time'>13:43</var> PST Resolved - Our upstream provider, Intercom, is back online. Customers can now utilize live chat to reach out for support. Jan <var data-var='date'> 9</var>, <var data-var='time'>13:10</var> PST Monitoring - Intercom has implemented a fix and we are monitoring the results. Live chat should be available but if you have any issues, please email us at customercare@lattice.com. Jan <var data-var='date'> 9</var>, <var data-var='time'>12:10</var> PST Investigating - Our upstream provider, Intercom, is currently experiencing an issue that prevents our customers from reaching Customer Care via live chat. If you need immediate assistance, please email us at customercare@lattice.com.

Live Chat Interruption

Dec <var data-var='date'> 5</var>, <var data-var='time'>01:30</var> PST Resolved - Cloudflare is reporting that a fix has been implemented and the issue has been resolved, and we're seeing the Lattice application load successfully. Our Engineering Team will continue to monitor the effects of this downtime on our services. Dec <var data-var='date'> 5</var>, <var data-var='time'>01:16</var> PST Investigating - An issue with an upstream provider, Cloudflare, is causing an outage of the Lattice platform. Our Engineering Team are aware of this issue, and are actively investigating the effects of this downtime on our services.

Dec <var data-var='date'> 3</var>, <var data-var='time'>17:49</var> PST Resolved - This incident has been resolved. Dec <var data-var='date'> 3</var>, <var data-var='time'>17:04</var> PST Identified - Admins may run into an error when dowloading review packets in a cycle. We have identified the issue, and the team is investigating a fix.

Error when downloading review packets

Nov <var data-var='date'>18</var>, <var data-var='time'>07:06</var> PST Resolved - Notifications are now being re-sent for any notifications affected. Our Engineering Team have identified that this was caused by the outages with Cloudflare and our email service provider. Nov <var data-var='date'>18</var>, <var data-var='time'>06:24</var> PST Investigating - We're aware of delays in the sending of Lattice notifications via email, Teams, and Slack. Our Engineering Team are investigating the root cause behind these delays, and will be re-sending any delayed notifications.

Is Lattice down?

Lattice status is UP

Lastest published events

Admin Compensation Cycle Recommendations page showing "Something Went Wrong" error

Lattice AI Agent and Evidence-Based Reviews generation errors

Issues sending email and Slack notifications

Lattice Services, Components and Regions

Recent Lattice events

Admin Compensation Cycle Recommendations page showing "Something Went Wrong" error

Lattice AI Agent and Evidence-Based Reviews generation errors

Issues sending email and Slack notifications

Issues logging in and completing actions in Lattice

Automated Review Cycle Schedule Delays

Slack Settings Deactivated for Some Users