Sep <var data-var='date'>18</var>, <var data-var='time'>11:57</var> UTC Resolved - We are processing real-time data for all data types again. Sep <var data-var='date'>18</var>, <var data-var='time'>11:45</var> UTC Update - Error ingestion is fully operational. We are currently working on recovering real-time processing of spans. Sep <var data-var='date'>18</var>, <var data-var='time'>10:58</var> UTC Identified - We have identified a networking issue with an upstream provider and are working on mitigations. Sep <var data-var='date'>18</var>, <var data-var='time'>10:56</var> UTC Update - We have identified a networking issue with an upstream provider and are working on mitigations. Sep <var data-var='date'>18</var>, <var data-var='time'>09:54</var> UTC Monitoring - We are processing real-time data again and are working on burning our backlog. Sep <var data-var='date'>18</var>, <var data-var='time'>09:53</var> UTC Update - We are processing real-time data again and are working on burning our backlog. Sep <var data-var='date'>18</var>, <var data-var='time'>09:00</var> UTC Update - We have implemented a mitigation and are starting to recover. Sep <var data-var='date'>18</var>, <var data-var='time'>08:36</var> UTC Investigating - We are currently investigating an issue that causes new errors to be delayed in the EU region.

Delayed ingestion of errors in EU

Sep <var data-var='date'>18</var>, <var data-var='time'>10:58</var> UTC Resolved - We have identified a networking issue with an upstream provider and are working on mitigations. Sep <var data-var='date'>18</var>, <var data-var='time'>10:56</var> UTC Update - We have identified a networking issue with an upstream provider and are working on mitigations. Sep <var data-var='date'>18</var>, <var data-var='time'>10:17</var> UTC Investigating - We are currently investigating an issue affecting spans ingestion in the EU region

Delays in spans ingestion

Sep <var data-var='date'>16</var>, <var data-var='time'>22:06</var> UTC Completed - The scheduled maintenance has been completed Sep <var data-var='date'>16</var>, <var data-var='time'>22:06</var> UTC Update - Scheduled maintenance is still in progress. We will provide updates as necessary. Sep <var data-var='date'>16</var>, <var data-var='time'>21:15</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>14</var>, <var data-var='time'>22:10</var> UTC Scheduled - We will be performing a database migration in the EU region that will pause span ingestion and disable span alerting for up to 2 hours. The Sentry UI and API should be available for the duration, but there will be a period of catch-up where ingestion will be behind and alerts will not fire (until ingestion has reached fresh data).

EU: Span ingestion temporarily delayed and alerting offline for database maintenance

Sep <var data-var='date'>10</var>, <var data-var='time'>17:10</var> UTC Resolved - Ingestion is now operating normally. Sep <var data-var='date'>10</var>, <var data-var='time'>16:52</var> UTC Update - We are continuing to monitor delayed ingestion. Sep <var data-var='date'>10</var>, <var data-var='time'>16:13</var> UTC Investigating - We are investigating degraded ingestion in US.

Ingestion is delayed in US

Sep <var data-var='date'>10</var>, <var data-var='time'>16:26</var> UTC Resolved - Issue fully resolved Sep <var data-var='date'>10</var>, <var data-var='time'>16:07</var> UTC Monitoring - Issue were identified and fixes were applied. We continue to monitor situation Sep <var data-var='date'>10</var>, <var data-var='time'>15:25</var> UTC Identified - During migration to new control silo region we started to observe increased number of 500 errors. Migration procedure was rolled back.

Sentry.io elevated number of 500 errors

Sep <var data-var='date'> 1</var>, <var data-var='time'>20:29</var> UTC Resolved - Spans ingestion is now operating normally. Sep <var data-var='date'> 1</var>, <var data-var='time'>19:31</var> UTC Update - Latency is almost fully recovered. Sep <var data-var='date'> 1</var>, <var data-var='time'>18:30</var> UTC Monitoring - A fix has been implemented, and latency is coming down. Sep <var data-var='date'> 1</var>, <var data-var='time'>17:50</var> UTC Update - We are continuing to investigate an issue that affects spans ingestion latency Sep <var data-var='date'> 1</var>, <var data-var='time'>16:41</var> UTC Update - We are continuing to investigate an issue that affects spans ingestion latency Sep <var data-var='date'> 1</var>, <var data-var='time'>15:40</var> UTC Update - We are continuing to investigate an issue that affects spans ingestion latency Sep <var data-var='date'> 1</var>, <var data-var='time'>14:16</var> UTC Investigating - We are experiencing slow ingestion for spans in US

Span ingestion is degraded in US

Sep <var data-var='date'> 1</var>, <var data-var='time'>11:21</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'> 1</var>, <var data-var='time'>09:03</var> UTC Update - We're monitoring the implemented fix Sep <var data-var='date'> 1</var>, <var data-var='time'>08:57</var> UTC Monitoring - We identified the issue and rolled out a configuration change Sep <var data-var='date'> 1</var>, <var data-var='time'>08:01</var> UTC Investigating - We are currently investigating what's going on and why requests timing out

Sentry requests timing out / 504's

Aug <var data-var='date'>31</var>, <var data-var='time'>22:43</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>31</var>, <var data-var='time'>21:26</var> UTC Update - Our systems are recovering from an ingestion backlog, webhooks are currently delayed Aug <var data-var='date'>31</var>, <var data-var='time'>21:16</var> UTC Monitoring - Our systems are recovering from an ingestion backlog Aug <var data-var='date'>31</var>, <var data-var='time'>21:12</var> UTC Investigating - API is experiencing issues Aug <var data-var='date'>31</var>, <var data-var='time'>21:02</var> UTC Monitoring - The fix has been deployed and our ingestion backlogs are recovering Aug <var data-var='date'>31</var>, <var data-var='time'>20:52</var> UTC Update - We have identified the culprit and are rolling out a fix Aug <var data-var='date'>31</var>, <var data-var='time'>20:31</var> UTC Investigating - we are investigating a failure in our infrastructure that is delaying ingestion

Ingestion backlog

Aug <var data-var='date'>27</var>, <var data-var='time'>15:41</var> UTC Resolved - Our latency is now back to normal. Incident resolved Aug <var data-var='date'>27</var>, <var data-var='time'>13:58</var> UTC Update - Our latency is now back to normal. We'll keep monitoring our systems a few more time before declaring the incident as resolved Aug <var data-var='date'>27</var>, <var data-var='time'>13:39</var> UTC Update - We keep monitoring our systems while we recover to our usual latency Aug <var data-var='date'>27</var>, <var data-var='time'>12:48</var> UTC Update - We are continuing to monitor while our systems recover after an issue that affected the availability of our product and API Aug <var data-var='date'>27</var>, <var data-var='time'>11:44</var> UTC Monitoring - We have identified and mitigated an issue that affected the availability of our product and API, and are now monitoring while our systems recover Aug <var data-var='date'>27</var>, <var data-var='time'>11:25</var> UTC Update - We are continuing to investigate an issue that affects the availability of our product and API Aug <var data-var='date'>27</var>, <var data-var='time'>10:49</var> UTC Update - We are continuing to investigate an issue that affects the availability of our product and API Aug <var data-var='date'>27</var>, <var data-var='time'>09:56</var> UTC Update - We are continuing to investigate an issue that affects the availability of our product and API Aug <var data-var='date'>27</var>, <var data-var='time'>09:53</var> UTC Update - We are continuing to investigate an issue that affects the availability of our product and API Aug <var data-var='date'>27</var>, <var data-var='time'>09:24</var> UTC Investigating - We are currently investigating an issue that affects the availability of our product and API

sentry.io is not available

Aug <var data-var='date'>26</var>, <var data-var='time'>20:36</var> UTC Resolved - The issue has been resolved Aug <var data-var='date'>26</var>, <var data-var='time'>20:04</var> UTC Update - We have put a fix into place and are monitoring the situation Aug <var data-var='date'>26</var>, <var data-var='time'>18:47</var> UTC Monitoring - We have put a fix into place and are monitoring the situation Aug <var data-var='date'>26</var>, <var data-var='time'>17:53</var> UTC Identified - We have identified a potential cause of excess traffic and are working on a fix. Aug <var data-var='date'>26</var>, <var data-var='time'>16:17</var> UTC Investigating - We are experiencing slower page loads and API responses due to a database issue in us

Issues performance is degraded in the US

Aug <var data-var='date'>25</var>, <var data-var='time'>12:38</var> UTC Resolved - Error alerting is back to normal. Aug <var data-var='date'>25</var>, <var data-var='time'>12:27</var> UTC Identified - We have identified the issue and implemented a fix. Aug <var data-var='date'>25</var>, <var data-var='time'>11:09</var> UTC Update - We are continuing to investigate the issue Aug <var data-var='date'>25</var>, <var data-var='time'>09:38</var> UTC Investigating - We are investigating delays in error alerts in the US region, up to 10 minutes

Errors alerting degraded in US region

Aug <var data-var='date'>25</var>, <var data-var='time'>03:14</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>25</var>, <var data-var='time'>03:07</var> UTC Verifying - Verification is currently underway for the maintenance items. Aug <var data-var='date'>25</var>, <var data-var='time'>02:59</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>19</var>, <var data-var='time'>16:37</var> UTC Scheduled - On August 24 @8pm PST Sentry will be performing maintenance on our central control plane. The purpose of this maintenance is to ensure the ongoing scalability and resilience of Sentry’s underlying systems. We anticipate this maintenance to take no more than two hours. During this maintenance our users may experience slowness with API calls and integrations. This will NOT impact the ingestion of events into Sentry’s system. We have chosen this time as it is between our peak periods during North American and European business hours, and represents the trough in events being ingested into Sentry’s system. Sentry will continually monitor systems throughout the entirety of the maintenance.

Control Plane Database Maintenance

Aug <var data-var='date'>18</var>, <var data-var='time'>22:21</var> UTC Resolved - The problem has been resolved, all backlogs have cleared and everything is operational again. Aug <var data-var='date'>18</var>, <var data-var='time'>22:15</var> UTC Monitoring - We have implemented a fix and the systems are recovering. We are monitoring to ensure that backlogs are proessed. Aug <var data-var='date'>18</var>, <var data-var='time'>21:56</var> UTC Identified - We have identified the cause and are working to determine a mitigation. This is only affecting US right now. Aug <var data-var='date'>18</var>, <var data-var='time'>21:42</var> UTC Investigating - We are currently investigating a delay in evaluating alerts in the US region.

Alert backlogs in US

Aug <var data-var='date'>13</var>, <var data-var='time'>22:27</var> UTC Resolved - All backlogs have been processed. Aug <var data-var='date'>13</var>, <var data-var='time'>22:13</var> UTC Update - We continue to make progress on the spans backlog with an estimated completion time of 15 minutes. Aug <var data-var='date'>13</var>, <var data-var='time'>21:22</var> UTC Update - We're continuing to make progress on the span ingestion backlog in US. Aug <var data-var='date'>13</var>, <var data-var='time'>20:32</var> UTC Update - We've recovered all ingestion latency in DE. In US, errors have processed their backlog and we continue to make progress on spans in US. Aug <var data-var='date'>13</var>, <var data-var='time'>19:53</var> UTC Update - The problematic change has been reverted and we're processing backlogs. Aug <var data-var='date'>13</var>, <var data-var='time'>19:34</var> UTC Monitoring - The problematic change has been reverted and we're processing backlogs. Aug <var data-var='date'>13</var>, <var data-var='time'>18:35</var> UTC Identified - We've identified the source of latency and are reverting that change. Aug <var data-var='date'>13</var>, <var data-var='time'>18:07</var> UTC Update - We are currently investigating ingestion delays for spans, transactions and alerts in DE and US Aug <var data-var='date'>13</var>, <var data-var='time'>17:59</var> UTC Update - We are currently investigating ingestion delays for spans and transactions in DE and US Aug <var data-var='date'>13</var>, <var data-var='time'>17:47</var> UTC Investigating - We are currently investigating ingestion delays for spans and transactions in DE

Span & Transaction ingestion delayed in DE

Aug <var data-var='date'>12</var>, <var data-var='time'>22:52</var> UTC Resolved - Backlogs have been processed Aug <var data-var='date'>12</var>, <var data-var='time'>22:43</var> UTC Identified - We've mitigated the source of ingestion latency and are processing the backlog. Aug <var data-var='date'>12</var>, <var data-var='time'>21:58</var> UTC Investigating - We are currently investigating error ingestion latency in US.

Error ingestion delays in US

Aug <var data-var='date'>12</var>, <var data-var='time'>14:48</var> UTC Resolved - The backlog has been processed Aug <var data-var='date'>12</var>, <var data-var='time'>14:21</var> UTC Monitoring - We're processing new events and alerts in realtime, and continue to process the tail of the backlog. Aug <var data-var='date'>12</var>, <var data-var='time'>13:28</var> UTC Identified - A fix has been made and the backlog is being processed. Aug <var data-var='date'>12</var>, <var data-var='time'>12:36</var> UTC Update - We've identified the root cause and are actively working on mitigation. Aug <var data-var='date'>12</var>, <var data-var='time'>11:45</var> UTC Investigating - We are actively investigating this issue.

Delayed Error ingestion in our US region

Aug <var data-var='date'>12</var>, <var data-var='time'>14:24</var> UTC Resolved - API performance has returned to expected levels. Aug <var data-var='date'>12</var>, <var data-var='time'>13:14</var> UTC Monitoring - API performance has returned to expected levels. We are continuing to monitor. Aug <var data-var='date'>12</var>, <var data-var='time'>12:58</var> UTC Investigating - We are currently investigating reports of API timeouts for the Sentry API in our US region.

Sentry API timeouts (US)

Aug <var data-var='date'> 7</var>, <var data-var='time'>17:08</var> UTC Resolved - The ingestion delay is resolved Aug <var data-var='date'> 7</var>, <var data-var='time'>16:48</var> UTC Monitoring - Ingestion latency for spans, crons, and logs is back to normal levels. Aug <var data-var='date'> 7</var>, <var data-var='time'>16:30</var> UTC Identified - We've put a fix in place and are monitoring. We anticipate catching up to ingestion within 1 hour Aug <var data-var='date'> 7</var>, <var data-var='time'>16:21</var> UTC Investigating - We are currently investigating the issue

Spans, crons, logs ingestion delayed in US region

Aug <var data-var='date'> 6</var>, <var data-var='time'>11:21</var> UTC Resolved - We've processed all backlogged events and have fully recovered. Aug <var data-var='date'> 6</var>, <var data-var='time'>10:50</var> UTC Monitoring - We're processing new events in real time, and are working through a backlog of events now. Aug <var data-var='date'> 6</var>, <var data-var='time'>10:25</var> UTC Update - We've identified the issue causing delays in ingestion and have applied a fix. We'll update again when we return to real time processing. Aug <var data-var='date'> 6</var>, <var data-var='time'>09:55</var> UTC Update - We are still actively investigating this issue. Aug <var data-var='date'> 6</var>, <var data-var='time'>08:59</var> UTC Investigating - We are actively investigating this issue.

Jul <var data-var='date'>27</var>, <var data-var='time'>18:27</var> UTC Resolved - The backlog has been cleared and ingestion is back to normal. Jul <var data-var='date'>27</var>, <var data-var='time'>18:07</var> UTC Monitoring - We have resolved the incident and are monitoring our ingestion backlog. Jul <var data-var='date'>27</var>, <var data-var='time'>16:56</var> UTC Identified - We have identified the issue and error and profile ingestion is back up in US. Expect delays in ingestion and alerting for errors and profiles. Jul <var data-var='date'>27</var>, <var data-var='time'>16:23</var> UTC Investigating - We are experiencing issues with error and profile ingestion in our US region and are investigating the issue.

Error and profile ingestion issues in US

Jul <var data-var='date'>21</var>, <var data-var='time'>14:14</var> UTC Resolved - We've burned through our spans backlog. Latency across span consumers are normal. Jul <var data-var='date'>21</var>, <var data-var='time'>13:55</var> UTC Update - We're still burning through our spans backlog Jul <var data-var='date'>21</var>, <var data-var='time'>13:04</var> UTC Monitoring - We've found and fixed the root cause and are burning through our spans backlog. Ingestion should be back to normal in around an hour. Jul <var data-var='date'>21</var>, <var data-var='time'>12:20</var> UTC Update - We are still actively investigating this issue Jul <var data-var='date'>21</var>, <var data-var='time'>11:26</var> UTC Update - We are still actively investigating this issue Jul <var data-var='date'>21</var>, <var data-var='time'>10:30</var> UTC Investigating - We are actively investigating this issue

Delayed ingestion on us region

Jul <var data-var='date'>20</var>, <var data-var='time'>16:57</var> UTC Resolved - We have fully burned the backlog of EU error events ingested during the incident. Latency is normal across all event types. Jul <var data-var='date'>20</var>, <var data-var='time'>16:09</var> UTC Update - We are still burning the backlog of EU region error events received during the incident. We are around halfway through the backlog. Jul <var data-var='date'>20</var>, <var data-var='time'>14:50</var> UTC Update - The backlog on attachments in our EU region is fully burned, and latency is back to normal. While new error events are being processed with normal latency, we have a backlog of old error events that we are now burning through. Jul <var data-var='date'>20</var>, <var data-var='time'>14:30</var> UTC Monitoring - We have implemented a fix for our attachment ingestion issue in our EU region, and are monitoring while the backlog burns. We will update here when latency is back to normal. Jul <var data-var='date'>20</var>, <var data-var='time'>13:47</var> UTC Update - We have implemented a fix for errors and feedback ingestion in our EU region, and latency for those pipelines has returned to normal. We are still working on fixing our attachments ingestion pipeline in EU. Jul <var data-var='date'>20</var>, <var data-var='time'>13:19</var> UTC Update - We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix. Jul <var data-var='date'>20</var>, <var data-var='time'>12:24</var> UTC Update - We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix. Jul <var data-var='date'>20</var>, <var data-var='time'>12:10</var> UTC Update - We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix. Jul <var data-var='date'>20</var>, <var data-var='time'>11:41</var> UTC Identified - We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are working on a fix. Jul <var data-var='date'>20</var>, <var data-var='time'>11:11</var> UTC Investigating - we are investigating an issue affecting errors ingestion in the DE/EU region

EU Region Error, Attachment, and Feedback latency

Jul <var data-var='date'>17</var>, <var data-var='time'>06:57</var> UTC Resolved - The webhook backlog is now resolved. Jul <var data-var='date'>17</var>, <var data-var='time'>05:49</var> UTC Update - The webhook backlog is processing. Jul <var data-var='date'>17</var>, <var data-var='time'>04:50</var> UTC Update - The webhook backlog is processing. Jul <var data-var='date'>17</var>, <var data-var='time'>03:49</var> UTC Update - The webhook backlog is processing. Estimated recovery is now 2-2.5 hours. Jul <var data-var='date'>17</var>, <var data-var='time'>02:44</var> UTC Update - The webhook backlog is processing. Jul <var data-var='date'>17</var>, <var data-var='time'>01:28</var> UTC Update - The webhook backlog is processing. Jul <var data-var='date'>17</var>, <var data-var='time'>00:44</var> UTC Update - The webhook backlog is being processed, ETA ~3 hours until fully caught up. Seer code reviews will run and linked Github issues and PRs will update incrementally. Jul <var data-var='date'>16</var>, <var data-var='time'>21:06</var> UTC Identified - The issue is mitigated and the webhook backlog is being processed. Seer code reviews will run and linked Github issues and PRs will update incrementally.

Github Integration Webhook processing delay

Jul <var data-var='date'>14</var>, <var data-var='time'>19:32</var> UTC Resolved - Feedback ingestion is back to realtime and the backlog has been processed. Jul <var data-var='date'>14</var>, <var data-var='time'>18:16</var> UTC Update - We are continuing to monitor ingestion, and a fix is being deployed to unblock feedback ingestion. Jul <var data-var='date'>14</var>, <var data-var='time'>17:40</var> UTC Monitoring - We are processing errors in realtime, and error backlogs have been processed. We're continuing to monitor the situation before restoring feedback ingestion. Jul <var data-var='date'>14</var>, <var data-var='time'>17:02</var> UTC Update - We are still working on recovery. New errors should be processed in real time, and we are working on recovering historical data. We are working to start allowing user feedback in DE. Jul <var data-var='date'>14</var>, <var data-var='time'>16:49</var> UTC Update - We are still working on recovery. New errors should be processed in real time, and we are working on recovering historical data. Jul <var data-var='date'>14</var>, <var data-var='time'>16:42</var> UTC Identified - We have identified the problem and are working on recovery. Jul <var data-var='date'>14</var>, <var data-var='time'>15:42</var> UTC Update - We continue to investigate the issue. Jul <var data-var='date'>14</var>, <var data-var='time'>14:44</var> UTC Investigating - We are currently investigating the issue.

Error ingestion degraded in DE region

Jul <var data-var='date'>13</var>, <var data-var='time'>14:25</var> UTC Resolved - Service has been restored. Jul <var data-var='date'>13</var>, <var data-var='time'>12:41</var> UTC Monitoring - Service has been restored and we are actively monitoring the situation Jul <var data-var='date'>13</var>, <var data-var='time'>12:24</var> UTC Identified - We identified the issue and restoring service. Jul <var data-var='date'>13</var>, <var data-var='time'>11:51</var> UTC Investigating - We are investigating ingestion delays.

Is Sentry down?

Sentry status is UP

Lastest published events

Delayed ingestion of errors in EU

Delays in spans ingestion

EU: Span ingestion temporarily delayed and alerting offline for database maintenance

Sentry Services, Components and Regions

Recent Sentry events

Delayed ingestion of errors in EU

Delays in spans ingestion

EU: Span ingestion temporarily delayed and alerting offline for database maintenance

Ingestion is delayed in US

Sentry.io elevated number of 500 errors

Span ingestion is degraded in US