Jun <var data-var='date'>15</var>, <var data-var='time'>09:44</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>15</var>, <var data-var='time'>09:33</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>15</var>, <var data-var='time'>09:33</var> UTC Update - We are continuing to investigate this issue. Jun <var data-var='date'>15</var>, <var data-var='time'>09:25</var> UTC Investigating - We are currently investigating an issue affecting customers in Europe. The application is unavailable and users may be experiencing server errors. We will provide updates as soon as we have more information.

Application unavailable

Jun <var data-var='date'>12</var>, <var data-var='time'>10:44</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>10:28</var> UTC Monitoring - Currently monitoring the fix applied - should be done shortly Jun <var data-var='date'>12</var>, <var data-var='time'>10:19</var> UTC Identified - Dashboard chart queries are erroring with "Invalid timezone: project_timezone" (for app.lightdash.cloud users) We've identified the issue and are working on a fix.

Dashboard charts failures (app.lightdash.cloud)

May <var data-var='date'> 9</var>, <var data-var='time'>09:09</var> UTC Completed - Success! Thank you for your patience! May <var data-var='date'> 9</var>, <var data-var='time'>08:21</var> UTC Update - Scheduled database maintanence. Lightdash may be temporarily unavailable for the next 25 minutes. May <var data-var='date'> 9</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'>29</var>, <var data-var='time'>11:11</var> UTC Scheduled - Lightdash Cloud EU will undergo scheduled maintenance on Saturday 9 May, 07:00–11:00 UTC. During this window, Lightdash Cloud's EU egress IP for outbound warehouse connections will move to a new static IP: 34.38.17.228. Customer impact: - Customers who allowlist Lightdash IPs in their warehouse, VPN, or firewall must add 34.38.17.228 to their allowlist before the window begins, in anticipation of the switchover. - Keep 34.79.239.130 and 34.140.90.83 allowlisted alongside the new IP. Do not remove them yet. We will post a follow-up here once it is safe to remove them. - At the moment of the cutover, active warehouse queries running through Lightdash will be interrupted and will need to be retried. The interruption is brief (under one minute). - Customers who do not allowlist Lightdash IPs are not affected. Updates will be posted here.

Action required: EU egress IP change

May <var data-var='date'> 6</var>, <var data-var='time'>16:55</var> UTC Resolved - Services stable, incident has been resolved. May <var data-var='date'> 6</var>, <var data-var='time'>15:42</var> UTC Monitoring - Systems are stable, fix has been applied successfully. Monitoring system vitals. May <var data-var='date'> 6</var>, <var data-var='time'>15:32</var> UTC Identified - A fix has been applied to the database and is rolling out. May <var data-var='date'> 6</var>, <var data-var='time'>15:14</var> UTC Investigating - We're investigating elevated database latency on Lightdash Cloud (US). Queries may be slower than usual. EU is unaffected.

Degraded database performance (US)

Apr <var data-var='date'>21</var>, <var data-var='time'>14:48</var> UTC Resolved - Earlier today, an additional egress IP address was unintentionally provisioned on our EU cluster. This caused some queries from Lightdash to customer warehouses to be blocked, where the new IP was not on the customer's allowlist. We've worked directly with affected customers to apply an allowlist update, and error rates have returned to normal. We're now monitoring to confirm stability and working with our cloud provider to consolidate back to a single egress IP at the infrastructure layer. Action required if you're still seeing warehouse connection errors: please ensure both `34.79.239.130` and `34.140.90.83` are on your warehouse IP allowlist. Our [IP allowlisting guide](https://docs.lightdash.com/get-started/setup-lightdash/connect-project#adding-lightdashs-static-ip-addresses-to-your-allow-list) has been updated to reflect the current EU egress IP set and includes step-by-step instructions. We apologise for the disruption. Apr <var data-var='date'>21</var>, <var data-var='time'>11:37</var> UTC Identified - We unintentionally added a new external IP address for customers in the EU. Customers using firewalls for their warehouse may see unexpected requests from: 34.140.90.83 - we are fixing this now.

Network configuration may cause warehouse timeout errors

Mar <var data-var='date'>24</var>, <var data-var='time'>12:19</var> UTC Resolved - We've seen no errors related to network configuration. Mar <var data-var='date'>24</var>, <var data-var='time'>11:34</var> UTC Monitoring - We have deployed a fix to our network configuration and all services are healthy. We will continue monitoring. Mar <var data-var='date'>24</var>, <var data-var='time'>10:36</var> UTC Update - We are continuing to work on a fix for this issue. Mar <var data-var='date'>24</var>, <var data-var='time'>10:35</var> UTC Update - We are continuing to work on a fix for this issue. Mar <var data-var='date'>24</var>, <var data-var='time'>10:35</var> UTC Identified - We have identified an issue in our network configuration that is leading to some requests failing. We've identified fix and are deploying to production.

Intermittent API errors across US

Sep <var data-var='date'>29</var>, <var data-var='time'>19:26</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'>29</var>, <var data-var='time'>19:25</var> UTC Update - We are continuing to monitor for any further issues. Sep <var data-var='date'>29</var>, <var data-var='time'>19:09</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'>29</var>, <var data-var='time'>18:34</var> UTC Identified - Some customers may notice embedded content not loading and the settings page missing. We've found an error with an external provider we use to flag this feature, we're working on a fix now

Embedding features may be unavailable

Jun <var data-var='date'>12</var>, <var data-var='time'>23:16</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>22:12</var> UTC Monitoring - Queries are loading. Waiting for GCP to confirm services are stable Jun <var data-var='date'>12</var>, <var data-var='time'>18:40</var> UTC Identified - Google Cloud Storage outage. Core services used to power Lightdash queries are currently unavailable. We're investigating work arounds. You can read more on Google Cloud Storage status here: https://status.cloud.google.com/

Google Cloud Platform outage

Jun <var data-var='date'>12</var>, <var data-var='time'>08:52</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>08:51</var> UTC Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>12</var>, <var data-var='time'>08:06</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>12</var>, <var data-var='time'>07:50</var> UTC Investigating - We've identified an issue with loading all dashboards on shared Lightdash instances.

Users can't access all dashboards (on app.lightdash.cloud)

Jun <var data-var='date'> 2</var>, <var data-var='time'>10:50</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'> 2</var>, <var data-var='time'>10:06</var> UTC Update - We're currently experiencing downtime on some of our larger instances, including: - app.lightdash.cloud - eu1.lightdash.cloud Our team is actively investigating the issue and working to restore access as quickly as possible. We’ll keep you updated as soon as we know more. Thanks for your patience! Jun <var data-var='date'> 2</var>, <var data-var='time'>09:43</var> UTC Investigating - We are currently investigating this issue.

Degraded performance

May <var data-var='date'>29</var>, <var data-var='time'>13:17</var> UTC Resolved - We can confirm that the fix was successful and that the issue has been resolved. Thank you for your patience May <var data-var='date'>29</var>, <var data-var='time'>12:57</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>29</var>, <var data-var='time'>11:28</var> UTC Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'>29</var>, <var data-var='time'>10:42</var> UTC Investigating - Users may notice slower page load times and delayed query responses during this time. Our engineering team is actively investigating the root cause and working to restore normal performance levels.

Degraded performance in app.lightdash.cloud

Apr <var data-var='date'>14</var>, <var data-var='time'>11:35</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'>14</var>, <var data-var='time'>10:33</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>14</var>, <var data-var='time'>09:11</var> UTC Update - We are continuing to investigate this issue. Apr <var data-var='date'>14</var>, <var data-var='time'>09:11</var> UTC Investigating - Users reporting degraded performance and timeout errors. We are investigating the issue.

Feb <var data-var='date'>10</var>, <var data-var='time'>18:06</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>10</var>, <var data-var='time'>17:12</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>10</var>, <var data-var='time'>15:48</var> UTC Identified - The issue has been identified and a fix is being implemented. Feb <var data-var='date'>10</var>, <var data-var='time'>15:48</var> UTC Investigating - Users are unable to complete the Google Sheets export functionality due to authentication failures during the sign-in process.

Google Sheets integration login issues

Sep <var data-var='date'>16</var>, <var data-var='time'>09:57</var> UTC Resolved - The email provider has resolved the incident, and email delivery is back to normal. Sep <var data-var='date'>16</var>, <var data-var='time'>09:39</var> UTC Identified - We've identified the cause of the email delivery issue—it's related to our email provider. They’ve updated their status page with more information and we're closely monitoring the situation and will provide further updates as they become available. Sep <var data-var='date'>16</var>, <var data-var='time'>09:31</var> UTC Investigating - We're currently experiencing an issue where emails are not being sent. Our team is actively investigating the problem.

Email Delivery Issue

Aug <var data-var='date'>12</var>, <var data-var='time'>18:01</var> UTC Resolved - Google has resolved the incident. More details can be found in their incident report: https://status.cloud.google.com/incidents/ETJGhvY9Xaktw7tgi8dF Aug <var data-var='date'>12</var>, <var data-var='time'>14:13</var> UTC Investigating - We are impacted by a broader issue from google affecting multiple Google services. The root cause is an outage within Google's infrastructure, which is beyond our control. You may experience issues with the following features: login with Google and Okta functionality, and Google Sheet Syncs. We are actively monitoring these services.

Google-related services impacted

Jun <var data-var='date'>18</var>, <var data-var='time'>05:10</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>17</var>, <var data-var='time'>20:50</var> UTC Monitoring - Emails are processing again, we are monitoring our external provider. Jun <var data-var='date'>17</var>, <var data-var='time'>19:03</var> UTC Identified - Users are not receiving emails from Lightdash, including verify emails on signup and password resets. We've identified that our email provider is experiencing an outage: https://status.postmarkapp.com/notices/zno1dlxjdjmblc0d-service-issue-outbound-sending-and-inbound-processing-messages-are-being-accepted-and-queued We're monitoring the status of our external provider while we setup a backup email service.

Email not sending

May <var data-var='date'> 1</var>, <var data-var='time'>14:06</var> UTC Resolved - This incident has been resolved May <var data-var='date'> 1</var>, <var data-var='time'>13:37</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'> 1</var>, <var data-var='time'>13:05</var> UTC Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'> 1</var>, <var data-var='time'>11:33</var> UTC Investigating - Users are experiencing unexpected errors. We are currently investigating the issue.

Degraded performance on US cloud instance

Mar <var data-var='date'> 6</var>, <var data-var='time'>18:51</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 6</var>, <var data-var='time'>17:29</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'> 6</var>, <var data-var='time'>16:49</var> UTC Identified - The issue has been identified and a fix is being implemented. Mar <var data-var='date'> 6</var>, <var data-var='time'>16:15</var> UTC Investigating - We are currently investigating the issue.

Degraded performance on cloud instances

Feb <var data-var='date'>28</var>, <var data-var='time'>18:42</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>28</var>, <var data-var='time'>17:54</var> UTC Monitoring - We've deployed a fix to all customers and are monitoring performance closely. Users are reporting normal loading times. Feb <var data-var='date'>28</var>, <var data-var='time'>17:09</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>28</var>, <var data-var='time'>17:06</var> UTC Update - We're currently deploying a fix to all customers, once this is deployed we'll begin monitoring the fix to ensure we've restored services to normal. Feb <var data-var='date'>28</var>, <var data-var='time'>16:34</var> UTC Identified - Dashboards and scheduled deliveries are experiencing degraded performance (>1 minute) load times due to a service outage with our feature flag provider (https://status.posthog.com) We're currently removing any calls to this service to return Lightdash to normal. Please subscribe for updates.

Performance degraded on dashboards and scheduled deliveries

Jan <var data-var='date'>29</var>, <var data-var='time'>17:08</var> UTC Resolved - This incident has been resolved by applying a fix and releasing a new version of the app. Thank you for your patience during this time. Jan <var data-var='date'>29</var>, <var data-var='time'>16:25</var> UTC Monitoring - A fix has been implemented and we're currently monitoring all instances. We recommend applying a "hard refresh" to your browser tab/window to ensure you get the latest changes applied Jan <var data-var='date'>29</var>, <var data-var='time'>16:24</var> UTC Update - We've identified the issue and are currently building a new image: 0.965.6 Jan <var data-var='date'>29</var>, <var data-var='time'>15:40</var> UTC Identified - We’ve identified an issue within the following release: 0.956.4, leading to empty Explores when deploying to a Lightdash project or creating Preview environments. We’ve reverted all Cloud instances. For self-hosted users, we recommend upgrading to version 0.965.6 in Lightdash.

Issues with deploying and creating preview environments

Oct <var data-var='date'>20</var>, <var data-var='time'>07:08</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>19</var>, <var data-var='time'>16:55</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Oct <var data-var='date'>19</var>, <var data-var='time'>15:56</var> UTC Update - We are continuing to work on a fix for this issue. Oct <var data-var='date'>19</var>, <var data-var='time'>15:55</var> UTC Identified - We identified the root cause as the database running at full capacity at the moment. We are going to scale the resources and identify what queries should be optimized. Oct <var data-var='date'>19</var>, <var data-var='time'>15:48</var> UTC Investigating - We are currently investigating this issue.

Degraded performance on app.lightdash.cloud

Oct <var data-var='date'> 3</var>, <var data-var='time'>15:39</var> UTC Resolved - We've successfully identified the root cause and have taken measures to resolve the issue. We apologize for any inconvenience this may have caused and appreciate your understanding and patience during this time.

Degraded performance and Snowflake Query Failures

Jun <var data-var='date'>27</var>, <var data-var='time'>22:03</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>27</var>, <var data-var='time'>16:00</var> UTC Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'>27</var>, <var data-var='time'>15:59</var> UTC Investigating - We've noticed degraded performance and disconnects for customers in the US region using app.lightdash.cloud. We're investigating the root cause

May <var data-var='date'> 9</var>, <var data-var='time'>15:33</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 9</var>, <var data-var='time'>15:22</var> UTC Monitoring - We are monitoring performance May <var data-var='date'> 9</var>, <var data-var='time'>13:48</var> UTC Investigating - The Lightdash API a https://app.lightdash.cloud is experiencing very slow response times for all users, leading to some actions taking minutes or not executing. We're currently investigating.

Lightdash Cloud instability (slow response times)

Jan <var data-var='date'>26</var>, <var data-var='time'>20:26</var> UTC Resolved - This incident is resolved. Jan <var data-var='date'>26</var>, <var data-var='time'>19:58</var> UTC Monitoring - A fix has been implemented and we are monitoring the deployment. Jan <var data-var='date'>26</var>, <var data-var='time'>19:49</var> UTC Identified - We've identified the root cause and it's being fixed. Jan <var data-var='date'>26</var>, <var data-var='time'>19:03</var> UTC Update - We have identified the failing component but are still finding the root cause. Services appear operational again. Jan <var data-var='date'>26</var>, <var data-var='time'>16:49</var> UTC Investigating - We are currently investigating the root cause of the issue

Is Lightdash down?

Lightdash status is UP

Lastest published events

Application unavailable

Dashboard charts failures (app.lightdash.cloud)

Action required: EU egress IP change

Lightdash Services, Components and Regions

Recent Lightdash events

Application unavailable

Dashboard charts failures (app.lightdash.cloud)

Action required: EU egress IP change

Degraded database performance (US)

Network configuration may cause warehouse timeout errors

Intermittent API errors across US