Apr <var data-var='date'> 7</var>, <var data-var='time'>23:42</var> UTC Resolved - Hot fix merged, all customers can access their subscription pages now.Apr <var data-var='date'> 7</var>, <var data-var='time'>22:28</var> UTC Monitoring - We are rolling out a fixApr <var data-var='date'> 7</var>, <var data-var='time'>21:36</var> UTC Identified - We have identified an issue for customers trying to access their subscription page. we are working on a fix

Subcription management page not working

Apr <var data-var='date'> 7</var>, <var data-var='time'>21:23</var> UTC Resolved - This incident has been resolvedApr <var data-var='date'> 7</var>, <var data-var='time'>21:02</var> UTC Monitoring - We performed a manual change that caused degraded performance and delayed ingestion in spans, crons, replays, and uptime that caused issues between 13:45 and 13:52 PT. The change has been cancelled and operations have returned to normal

US delays and ingestion and issues in querying across pipelines

Apr <var data-var='date'> 6</var>, <var data-var='time'>21:26</var> UTC Resolved - This incident has been resolved and backlogs have been processed.Apr <var data-var='date'> 6</var>, <var data-var='time'>20:52</var> UTC Monitoring - We have identified the source of postgres load and mitigated it. We are continuing to monitor conditions as backlogs process.Apr <var data-var='date'> 6</var>, <var data-var='time'>20:04</var> UTC Update - We are continuing to mitigate load in our postgres clusters.Apr <var data-var='date'> 6</var>, <var data-var='time'>19:04</var> UTC Update - We are continuing to mitigate load in our postgres clusters.Apr <var data-var='date'> 6</var>, <var data-var='time'>18:26</var> UTC Update - We're continuing to mitigate load in our caching and postgres infrastructure.Apr <var data-var='date'> 6</var>, <var data-var='time'>17:11</var> UTC Identified - We have identified the problem as a load issue in our caching layer and are working to mitigate the issue.Apr <var data-var='date'> 6</var>, <var data-var='time'>15:56</var> UTC Investigating - We are investigating processing delays for errors and spans in the US region

Error & Span processing delays in US

Mar <var data-var='date'>21</var>, <var data-var='time'>01:09</var> UTC Resolved - Ingestions and alerts are caught up and the incident is resolved.Mar <var data-var='date'>20</var>, <var data-var='time'>21:42</var> UTC Monitoring - Customers may experience some delay in ingestion and alerts. We are currently burning the backlog

Ingestion and alerts delayed in US

Mar <var data-var='date'>16</var>, <var data-var='time'>22:21</var> UTC Resolved - The incident has been resolvedMar <var data-var='date'>16</var>, <var data-var='time'>22:11</var> UTC Monitoring - Ingestion is back to normal, alerts may still be delayedMar <var data-var='date'>16</var>, <var data-var='time'>21:39</var> UTC Identified - We are actively mitigating the issueMar <var data-var='date'>16</var>, <var data-var='time'>20:17</var> UTC Investigating - We are currently investigating the issue. Some alerts may be dropped if they are for older time windows.

Ingestion delayed for transactions, uptime, spans, crons in US

Mar <var data-var='date'> 9</var>, <var data-var='time'>20:37</var> UTC Resolved - Ingestion and alert latency is back to normal.Mar <var data-var='date'> 9</var>, <var data-var='time'>18:17</var> UTC Monitoring - We are burning the backlog and actively monitoring the progress.Mar <var data-var='date'> 9</var>, <var data-var='time'>17:59</var> UTC Identified - US customers may experience delay in alerts. We've identified the issue and will be putting in a fix.

Alerts delayed for US customers

Mar <var data-var='date'> 2</var>, <var data-var='time'>21:09</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'> 2</var>, <var data-var='time'>20:57</var> UTC Update - EU ingestion has been restored and latency is back to normal levels. US continues to recover and will likely be caught up within the next hour.Mar <var data-var='date'> 2</var>, <var data-var='time'>20:29</var> UTC Monitoring - We have implemented a fix and are monitoringMar <var data-var='date'> 2</var>, <var data-var='time'>20:09</var> UTC Identified - The issue has been identified and the fix is being implemented.Mar <var data-var='date'> 2</var>, <var data-var='time'>19:57</var> UTC Investigating - We are currently investigating this issue.

Ingestion delays in US and EU regions

Feb <var data-var='date'>26</var>, <var data-var='time'>20:56</var> UTC Resolved - Ingestion backlog has finished processing and our system is now operating normally.Feb <var data-var='date'>26</var>, <var data-var='time'>20:02</var> UTC Monitoring - Our cloud provider has resolved an underlying problem and our dashboard availability issues have been resolved. We're continuing to process our ingestion backlog and monitor the situation.Feb <var data-var='date'>26</var>, <var data-var='time'>19:15</var> UTC Investigating - We're investigating intermittent failures loading our dashboard (all regions) and increased latency for ingestion of all events types in US

Intermittent dashboard failures & increased US ingest latency

Feb <var data-var='date'>26</var>, <var data-var='time'>19:41</var> UTC Resolved - We have identified that the core problem is related to the Intermittent dashboard failures. Please follow https://status.sentry.io/incidents/z3g2bjxxwv9l for the latest updates. In the meantime, this will be marked as resolved.Feb <var data-var='date'>26</var>, <var data-var='time'>19:07</var> UTC Update - We also identified that transaction ingestion was also affected.Feb <var data-var='date'>26</var>, <var data-var='time'>18:59</var> UTC Investigating - We are experiencing an ingestion issue with spans, logs, and metrics. Our teams are currently investigating the problem.

Ingestion Issue in US

Feb <var data-var='date'>24</var>, <var data-var='time'>13:31</var> UTC Resolved - This incident has been resolved. Ingestion in our South America point-of-presence is fully functional.Feb <var data-var='date'>24</var>, <var data-var='time'>13:00</var> UTC Investigating - We are currently investigating the issue.

Increased error rate in South America ingestion region

Feb <var data-var='date'>22</var>, <var data-var='time'>04:36</var> UTC Resolved - This incident has been resolved.Feb <var data-var='date'>22</var>, <var data-var='time'>04:32</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Feb <var data-var='date'>22</var>, <var data-var='time'>04:21</var> UTC Identified - Due to a transient error, some calls to Seer are failing in EU. We identified the issue and are in the process of deploying a fix.

Calls to Seer in DE are failing

Feb <var data-var='date'>13</var>, <var data-var='time'>00:57</var> UTC Resolved - We have resolved the issue, and our system is now working as expectedFeb <var data-var='date'>13</var>, <var data-var='time'>00:38</var> UTC Update - Ingestion is back to normal, alerts may still be delayed. We will continue to monitor the recovery.Feb <var data-var='date'>12</var>, <var data-var='time'>23:44</var> UTC Update - We are still continuing to process the backlog and are still monitoring the recovery.Feb <var data-var='date'>12</var>, <var data-var='time'>21:35</var> UTC Update - We are still continuing to process the backlog and are still monitoring the recovery.Feb <var data-var='date'>12</var>, <var data-var='time'>19:17</var> UTC Update - We are still continuing to process the backlog and are still monitoring the recovery.Feb <var data-var='date'>12</var>, <var data-var='time'>16:54</var> UTC Update - We are continuing to process the backlog and are still monitoring the recovery.Feb <var data-var='date'>12</var>, <var data-var='time'>15:00</var> UTC Monitoring - We've identified and fixed the issue. We are burning the backlog and actively monitoring the progress.Feb <var data-var='date'>12</var>, <var data-var='time'>09:43</var> UTC Investigating - We are investigating the resurfaced Zookeeper issueFeb <var data-var='date'>12</var>, <var data-var='time'>08:34</var> UTC Identified - We identified a problem with our Zookeeper cluster and fixed it. We now need to ingest the backlog.Feb <var data-var='date'>12</var>, <var data-var='time'>05:34</var> UTC Investigating - We are currently investigating the issue

Ingestion delays for spans, logs, traces, and metrics in US

Feb <var data-var='date'> 9</var>, <var data-var='time'>19:39</var> UTC Resolved - The incident has been resolvedFeb <var data-var='date'> 9</var>, <var data-var='time'>19:33</var> UTC Investigating - We are currently investigating

EU explore page performance degraded

Jan <var data-var='date'>29</var>, <var data-var='time'>23:43</var> UTC Resolved - We have resolved the issue and all systems are working as expected.Jan <var data-var='date'>29</var>, <var data-var='time'>23:24</var> UTC Monitoring - We have identified an issue related to database contention and have issued a fix. We are continuing to monitor the system as it returns to health.Jan <var data-var='date'>29</var>, <var data-var='time'>23:05</var> UTC Update - We are investigating an issue where the Sentry Dashboard may be slow to load.Jan <var data-var='date'>29</var>, <var data-var='time'>23:02</var> UTC Investigating - We are currently investigating this issue.

Investigating Issues with Dashboard

Jan <var data-var='date'>27</var>, <var data-var='time'>18:56</var> UTC Resolved - Ingestion has recovered.Jan <var data-var='date'>27</var>, <var data-var='time'>18:03</var> UTC Identified - The issue has been identified and a fix is being implemented.Jan <var data-var='date'>27</var>, <var data-var='time'>14:51</var> UTC Investigating - We are investigating a delay in ingesting spans in the US region.

Delay in span ingestion

Jan <var data-var='date'>24</var>, <var data-var='time'>00:27</var> UTC Resolved - Backlog has been processed and crons should be running as normalJan <var data-var='date'>23</var>, <var data-var='time'>21:46</var> UTC Identified - The issue has been identified and we working through backlogs.Jan <var data-var='date'>23</var>, <var data-var='time'>18:38</var> UTC Investigating - We are currently investigating this issue.

Crons ingestion backlogged

Jan <var data-var='date'>14</var>, <var data-var='time'>00:00</var> UTC Completed - The scheduled maintenance has been completed.Jan <var data-var='date'> 5</var>, <var data-var='time'>16:55</var> UTC Update - We completed the brownout successfully. The involved DNS records are back to the previous addresses and the schedule for the final switch has been confirmed for Tuesday, January 13th, 08:00-19:00 UTC.Jan <var data-var='date'> 5</var>, <var data-var='time'>00:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Nov <var data-var='date'>20</var>, <var data-var='time'>09:35</var> UTC Scheduled - We're going to update the IP addresses of the following ingestion DNS records: - *.ingest.sentry.io (currently pointing to 34.120.195.249/32) - *.ingest.us.sentry.io (currently pointing to 34.120.195.249/32) - *.ingest.de.sentry.io (currently pointing to 34.120.62.213/32) to a new unified set of IPv4 and IPv6 addresses, specifically: - 34.160.81.0/32 - 34.102.210.18/32 - 2600:1901:0:5e8a::/64 - 2600:1901:0:7edb::/64 A brownout has been scheduled for Monday, January 5th, 15:30-16:00 UTC, where both the old IPs and the new ones will be used, on all DNS records: - *.ingest.sentry.io - *.ingest.de.sentry.io - *.ingest.us.sentry.io The switch has been scheduled for Tuesday, January 13th, 08:00-19:00 UTC on all DNS records: - *.ingest.sentry.io - *.ingest.de.sentry.io - *.ingest.us.sentry.io We advise customers to add the new IP addresses to any filtering/whitelisting policy setup in their network configuration.

Ingestion IP addresses updates

Jan <var data-var='date'>12</var>, <var data-var='time'>17:56</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'>12</var>, <var data-var='time'>16:53</var> UTC Monitoring - Our UI has recovered from intermittent availability issues due to an increase in load on our DB, the load has recovered and the UI shouldn't experience any further issues. We're continuing to monitor.Jan <var data-var='date'>12</var>, <var data-var='time'>16:10</var> UTC Investigating - Our UI is unavailable again, we're investigating the issue.Jan <var data-var='date'>12</var>, <var data-var='time'>15:44</var> UTC Monitoring - The UI availability issue has been resolved, we're continuing to monitor the situation.Jan <var data-var='date'>12</var>, <var data-var='time'>15:30</var> UTC Investigating - We are currently investigating this issue.

UI unavailable

Jan <var data-var='date'>12</var>, <var data-var='time'>15:16</var> UTC Resolved - All backlogs have been processed and no data was lost.Jan <var data-var='date'>12</var>, <var data-var='time'>14:21</var> UTC Update - We are now ingesting recent transaction data again and are processing old data in the background.Jan <var data-var='date'>12</var>, <var data-var='time'>14:16</var> UTC Update - We have implemented a fix and are monitoring the situation closely. No data has been lost.Jan <var data-var='date'>12</var>, <var data-var='time'>13:33</var> UTC Investigating - We are investigating an issue affecting transaction ingestion in the EU region

Ingestion delays for transactions in EU regions

Jan <var data-var='date'> 8</var>, <var data-var='time'>22:21</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'> 8</var>, <var data-var='time'>22:04</var> UTC Monitoring - A fix has been implemented and we are processing the backlog that formed during the incident.Jan <var data-var='date'> 8</var>, <var data-var='time'>21:46</var> UTC Identified - We have identified the issue and a fix is being deployed.Jan <var data-var='date'> 8</var>, <var data-var='time'>20:47</var> UTC Investigating - We are currently investigating a delay in processing spans in the US region.

Processing delays for spans in US

Dec <var data-var='date'>30</var>, <var data-var='time'>20:04</var> UTC Resolved - The issue has been resolved, and services are operating normally.Dec <var data-var='date'>30</var>, <var data-var='time'>17:50</var> UTC Monitoring - A fix has been implemented, and we are monitoring the results.Dec <var data-var='date'>30</var>, <var data-var='time'>16:15</var> UTC Investigating - We are investigating increased request timeouts and 50x errors across multiple API endpoints that may be impacting service availability.

Increased request timeouts and 50x errors across multiple API endpoints

Dec <var data-var='date'>23</var>, <var data-var='time'>02:46</var> UTC Resolved - We had a database corruption where new org creation, user sign-ups, new API Token generation, and new integrations made between 3:45PM and 5:00PM PST were affected and will need to be re-created. No other data were affected. We are very sorry for this inconvenience.Dec <var data-var='date'>23</var>, <var data-var='time'>01:51</var> UTC Update - We restored the impacted services to operational state, while investigating the full impact of the incidentDec <var data-var='date'>23</var>, <var data-var='time'>01:32</var> UTC Update - We are continuing to monitor for any further issues.Dec <var data-var='date'>23</var>, <var data-var='time'>01:03</var> UTC Monitoring - We implemented a fix for the issue and monitoringDec <var data-var='date'>23</var>, <var data-var='time'>00:56</var> UTC Identified - We're implementing a fix for the issueDec <var data-var='date'>23</var>, <var data-var='time'>00:49</var> UTC Update - We detected a data corruption in one of our databases and investigating the issueDec <var data-var='date'>23</var>, <var data-var='time'>00:42</var> UTC Investigating - We are currently investigating this issue.

Database data corruption

Dec <var data-var='date'>20</var>, <var data-var='time'>08:34</var> UTC Resolved - This incident has been resolved.Dec <var data-var='date'>20</var>, <var data-var='time'>07:18</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Dec <var data-var='date'>20</var>, <var data-var='time'>06:40</var> UTC Identified - We have identified the cause of our current login issues and are working on a fix.Dec <var data-var='date'>20</var>, <var data-var='time'>06:38</var> UTC Update - We are continuing to investigate the issue related to logging in to the Sentry frontend in our US and DE regions.Dec <var data-var='date'>20</var>, <var data-var='time'>05:14</var> UTC Investigating - We are investigating an issue with logging in to the Sentry frontend.

Sentry Login Issues

Dec <var data-var='date'>18</var>, <var data-var='time'>15:05</var> UTC Resolved - Our UI was unavailable between 14:20 UTC and 14:23 UTC due to a bad configuration change, we have reverted the change and service has been restored.

UI was unavailable between 14:20 and 14:23 UTC

Dec <var data-var='date'>10</var>, <var data-var='time'>22:43</var> UTC Resolved - This incident has been resolved.Dec <var data-var='date'>10</var>, <var data-var='time'>21:47</var> UTC Update - We're continuing to process the backlog of errors and transactions. New events will be processed in a timely fashion.Dec <var data-var='date'>10</var>, <var data-var='time'>20:40</var> UTC Monitoring - We are continuing to work through backlogs in errors and transactions. Profiles, attachments and replays have recovered.Dec <var data-var='date'>10</var>, <var data-var='time'>20:10</var> UTC Update - Ingestion has resumed and we are continuing to monitor and restore full service.Dec <var data-var='date'>10</var>, <var data-var='time'>19:51</var> UTC Identified - We have identified the source of the issue and are scaling up infrastructure to restore serviceDec <var data-var='date'>10</var>, <var data-var='time'>19:22</var> UTC Update - We're investigating an unavailable system in our ingestion pipelines.Dec <var data-var='date'>10</var>, <var data-var='time'>19:18</var> UTC Investigating - We are currently investigating ingestion delays for errors, spans, attachments, profiles and replays in US.

Is Sentry down?

Sentry status is UP

Lastest published events

Subcription management page not working

US delays and ingestion and issues in querying across pipelines

Error & Span processing delays in US

Sentry Services, Components and Regions

Recent Sentry events

Subcription management page not working

US delays and ingestion and issues in querying across pipelines

Error & Span processing delays in US

Ingestion and alerts delayed in US

Ingestion delayed for transactions, uptime, spans, crons in US

Alerts delayed for US customers

Get your FREE account now,
5 monitors included!

Is Sentry down?

Sentry status is UP

Lastest published events

Subcription management page not working

US delays and ingestion and issues in querying across pipelines

Error & Span processing delays in US

Sentry Services, Components and Regions

Recent Sentry events

Subcription management page not working

US delays and ingestion and issues in querying across pipelines

Error & Span processing delays in US

Ingestion and alerts delayed in US

Ingestion delayed for transactions, uptime, spans, crons in US

Alerts delayed for US customers

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!