Aug <var data-var='date'>15</var>, <var data-var='time'>13:48</var> PDT Resolved - Between 19:54 and 20:07 UTC on August 15, requests to our US error reporting API (api.honeybadger.io) experienced an elevated failure rate. During this window, a majority of requests to submit errors and events timed out or received 5xx responses. Browser (JavaScript) error reporting, source map uploads, the Honeybadger web application, dashboards, uptime monitoring, and our EU region were not affected. What happened: Beginning at 19:54 UTC, a single source began submitting error reports at roughly 150 times its normal rate, with payloads several times larger than average. This more than doubled the total volume of data arriving at our ingestion tier, which saturated within a minute. Because the saturated service kept accepting connections rather than returning errors outright, our autoscaling and alerting — which watch for error responses — did not register the problem quickly, and capacity was added later than it should have been. Service recovered at 20:07 UTC once additional capacity came online. Most affected reports were retried successfully by our client libraries once the API recovered, so the large majority of data submitted during the window was ultimately received. Reports sent by libraries without retry logic were not received and cannot be recovered. What we've done: our ingestion tier now scales on response latency in addition to error rates, so this kind of saturation is detected within a minute or two rather than only once errors appear. We've also raised the tier's baseline capacity and widened our paging alerts to cover failures that don't surface as server errors. We're additionally adding limits on how much data a single source can submit. We're sorry for the disruption, and for any gaps this caused in your error data during the window.

Elevated failure rates on the US error reporting API

Jul <var data-var='date'> 8</var>, <var data-var='time'>04:48</var> PDT Resolved - Between 06:22 and 08:23 UTC on July 8, requests to our US error reporting API (api.honeybadger.io) experienced a highly elevated failure rate. During this window, a majority of requests to submit errors, events, and check-ins received 5xx responses or timed out. Error reports sent by client libraries without retry logic during this period were not received and cannot be recovered. The Honeybadger web application, dashboards, uptime monitoring, and our EU region were not affected. What happened: Beginning at 06:18 UTC, we started receiving error reports at roughly triple our normal total API volume. The service tier that ingests this traffic was at its overnight capacity level and became saturated. Our autoscaling responded, but the metrics it relied on significantly under-measured the failed traffic, so capacity was added far too slowly — full recovery came at 08:23 UTC when scaling caught up and absorbed the load. Compounding the problem, PagerDuty alerts were not fired to wake up a human due to a misconfiguration. What we've done: We've fixed the underlying scaling weakness: API capacity now also scales directly on load-balancer error rates (previously we scaled only on request rates, and that scaling doesn't respond quickly enough), and we've doubled the maximum capacity of the ingestion tier. These changes are already deployed to both US and EU regions. We're also evaluating adding adaptive per-key rate limiting to the ingestion service. We're sorry for the disruption, and for any gaps this caused in your error data during the window.

Elevated error rates on the US error reporting API

Mar <var data-var='date'>12</var>, <var data-var='time'>12:27</var> PDT Resolved - We had a backlog in our notifications queue that prevented timely delivery of alerts to channels such as Email, Slack, etc. The backlog has been processed as of 19:20 UTC. Our autoscaling rules for this service had a monitoring gap that prevented a scale-out to maintain a low latency on the queue. This monitoring gap has been resolved.

Delays with notifications

Jan <var data-var='date'>14</var>, <var data-var='time'>12:48</var> PST Resolved - This incident has been resolved. Jan <var data-var='date'>14</var>, <var data-var='time'>12:40</var> PST Identified - A recent deploy has resulted in our rejecting some check-in API requests. We are in the process of deploying a fix.

Outage in check-in processing

Oct <var data-var='date'>22</var>, <var data-var='time'>13:15</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>22</var>, <var data-var='time'>12:46</var> PDT Identified - The issue has been identified and a fix is being implemented.

Insights queries and dashboards are unavailable

Oct <var data-var='date'>22</var>, <var data-var='time'>06:16</var> PDT Resolved - The EU API has been restored. Oct <var data-var='date'>22</var>, <var data-var='time'>06:11</var> PDT Monitoring - The US API has been restored, and the EU API restoration is in process. Oct <var data-var='date'>22</var>, <var data-var='time'>06:09</var> PDT Identified - API ingestion of errors has been interrupted. We are deploying a fix.

Error ingestion API is down

Oct <var data-var='date'>20</var>, <var data-var='time'>12:58</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>12:58</var> PDT Monitoring - Processing has been restored and all systems are normal. We'll continue to monitor as AWS continues to restore service. Thanks! Oct <var data-var='date'>20</var>, <var data-var='time'>10:55</var> PDT Update - We're continuing to monitor the AWS outage. Errors are being processed, but may be delayed. Uptime checks are also delayed, so you may notice lag with uptime notifications and status page updates. Oct <var data-var='date'>20</var>, <var data-var='time'>06:44</var> PDT Identified - Our error processing pipeline has been impacted by the AWS outage earlier this morning. Errors are being processed, but the end-to-end latency is much higher than normal. We are working to restore the processing to full capacity.

Delays in error processing

Oct <var data-var='date'>20</var>, <var data-var='time'>01:00</var> PDT Resolved - Our systems were affected by the DynamoDB outage in AWS' us-east-1 region. Impacts included not being able to log use the web app, uptime checks not being executed on time, and check-ins being falsely reported as missing. We are evaluating what steps we can take to mitigate these kinds of impacts from this kind of outage in the future. Additionally, our status page was not updated in a timely manner due to our not having configured various AWS services as dependent components of our status page. We have fixed that issue, so future AWS outages will result in our status page automatically being updated.

AWS outage in us-east-1

Feb <var data-var='date'>12</var>, <var data-var='time'>05:30</var> PST Resolved - We had an interruption in processing of check-in alerts and in-app stats from about 12:30 to about 13:30 UTC. This may have resulted in a failure or a delay in reporting check-ins that went missing during that period.

Delay in stats and check-in processing

Dec <var data-var='date'>20</var>, <var data-var='time'>06:05</var> PST Completed - The scheduled maintenance has been completed. Some check-ins may have falsely reported as being missing during this window. We will work on improving that for future maintenance events. Dec <var data-var='date'>20</var>, <var data-var='time'>06:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>19</var>, <var data-var='time'>14:31</var> PST Scheduled - We will have a few minutes of app unavailability and error ingestion delays during a database upgrade.

Database maintenance

Oct <var data-var='date'>22</var>, <var data-var='time'>08:30</var> PDT Resolved - The backlog has been processed and all systems are running normally. Oct <var data-var='date'>22</var>, <var data-var='time'>06:35</var> PDT Monitoring - We have identified and corrected an issue in our data processing pipeline that may have caused delays for some customers. No data has been lost and the system should be caught up shortly.

Data Processing Delays

Sep <var data-var='date'>23</var>, <var data-var='time'>04:57</var> PDT Resolved - This incident has been resolved. Sep <var data-var='date'>23</var>, <var data-var='time'>04:48</var> PDT Monitoring - We have identified the cause of the slowdown and performance is back to normal. Sep <var data-var='date'>23</var>, <var data-var='time'>03:37</var> PDT Investigating - We are looking into the cause of slowness and timeouts with the web app.

App slowness and timeouts

Jul <var data-var='date'>24</var>, <var data-var='time'>20:06</var> PDT Resolved - The backlog has been fully processed, so all Insights events should be ingested and available. Jul <var data-var='date'>24</var>, <var data-var='time'>18:55</var> PDT Monitoring - We have successfully pushed a fix and are waiting for the backlog to be processed. I also want to give a reminder that this only affects Insights events; error processing was unaffected. Jul <var data-var='date'>24</var>, <var data-var='time'>16:42</var> PDT Identified - We have identified an issue with Insights event ingestion backlog. We are working on a fix.

Insights backlog issue

Feb <var data-var='date'>12</var>, <var data-var='time'>12:39</var> PST Resolved - We discovered and addressed a configuration issue that caused the delay in processing. Processing has returned to normal. Feb <var data-var='date'>12</var>, <var data-var='time'>12:22</var> PST Investigating - We are seeing an increased backlog in our check-in processing, which is causing false alarms to be reported. We are investigating the cause of the backlog.

Check-in monitoring

Feb <var data-var='date'> 6</var>, <var data-var='time'>15:19</var> PST Resolved - This incident has been resolved. Feb <var data-var='date'> 6</var>, <var data-var='time'>15:03</var> PST Monitoring - We've resolved an issue with query contention, and web application service has been restored. Our API pipeline was unaffected by this incident. Some charts on the reports tab may be outdated until we update project counts. We'll continue to monitor the situation. Sorry for the inconvenience! Feb <var data-var='date'> 6</var>, <var data-var='time'>13:42</var> PST Identified - We've identified an issue with aggregate database queries and are working on a fix. Feb <var data-var='date'> 6</var>, <var data-var='time'>13:02</var> PST Update - We are continuing to investigate this issue. Feb <var data-var='date'> 6</var>, <var data-var='time'>12:59</var> PST Investigating - We are currently experiencing elevated response times and timeouts from our web application.

Increased response times and timeouts

Jan <var data-var='date'>17</var>, <var data-var='time'>11:21</var> PST Resolved - This incident has been resolved. Jan <var data-var='date'>17</var>, <var data-var='time'>11:19</var> PST Identified - We are working on restoring service

Web app is unavailable

Nov <var data-var='date'>10</var>, <var data-var='time'>11:44</var> PST Resolved - We discovered this morning that our Logplex pipeline, which handles Heroku platform errors, had an interruption in processing, which caused platform errors not to be recorded. This has been corrected, the backlog is being processed, and additional monitoring has been added to avoid this issue in the future.

Delay in Logplex processing

Jun <var data-var='date'>13</var>, <var data-var='time'>14:37</var> PDT Resolved - Lambda is back in business, and the backlog of events triggered during the outage is being processed. Jun <var data-var='date'>13</var>, <var data-var='time'>12:34</var> PDT Update - Search indexing and notice timeline charts are also impacted by the failures with Lamba. We're continuing to queue updates for processing once Lambda resumes normal operations. Jun <var data-var='date'>13</var>, <var data-var='time'>11:59</var> PDT Identified - Issues with AWS Lambda are causing our uptime check reports to be delayed. Reports are queued up and will be delivered as AWS services recover.

Uptime check reports are delayed

Apr <var data-var='date'> 8</var>, <var data-var='time'>07:30</var> PDT Completed - The scheduled maintenance has been completed. Apr <var data-var='date'> 8</var>, <var data-var='time'>07:01</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'> 6</var>, <var data-var='time'>14:45</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Mar <var data-var='date'>24</var>, <var data-var='time'>15:30</var> PDT Resolved - Starting around 3:30pm PST, we had an incident where an unhealthy API server was in rotation in our load balancer. During that time requests routed to that server responded with a 502 which could have resulted in the clients dropping notices. At around 5:30pm PST we removed the unhealthy server from rotation. 7:22pm PST edit: Reworded message to describe that notices are not queued on the client, but dropped when the server responds in an error state

Unhealthy API server

Mar <var data-var='date'>24</var>, <var data-var='time'>02:01</var> PDT Resolved - We identified that our workers had an issue deploying. We deployed a fix and upped our capacity to deal with the current backlog. Everything is back to normal now. Turns out this issue was related GitHub rotating their public RSA key: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ Mar <var data-var='date'>24</var>, <var data-var='time'>01:40</var> PDT Identified - We have identified the issue and are working towards clearing the backlog. Mar <var data-var='date'>24</var>, <var data-var='time'>01:05</var> PDT Investigating - We currently are having issues processing our notice backlog. We are looking into the issue.

Issues processing backlog

Mar <var data-var='date'> 9</var>, <var data-var='time'>01:40</var> PST Resolved - We had an incident (from 1:00 to 1:27 PST) where our apps were unresponsive. They have since recovered.

Web app unavailable

Feb <var data-var='date'>14</var>, <var data-var='time'>08:22</var> PST Resolved - Our primary redis instance was having a bad time. This has been fixed, and now we're back to normal. Feb <var data-var='date'>14</var>, <var data-var='time'>08:10</var> PST Investigating - We are seeing increased error rates for our API

API issues

Dec <var data-var='date'>16</var>, <var data-var='time'>15:07</var> PST Resolved - This incident has been resolved. Dec <var data-var='date'>16</var>, <var data-var='time'>14:21</var> PST Update - We are continuing to monitor for any further issues. Dec <var data-var='date'>16</var>, <var data-var='time'>11:19</var> PST Monitoring - Our email provider is having problems delivering emails, so our outbound email deliveries are being delayed.

Email delivery is being delayed

Nov <var data-var='date'>23</var>, <var data-var='time'>01:23</var> PST Resolved - This incident has been resolved. Nov <var data-var='date'>23</var>, <var data-var='time'>01:17</var> PST Investigating - We are currently investigating this issue.

Is Honeybadger down?

Honeybadger status is UP

Lastest published events

Elevated failure rates on the US error reporting API

Elevated error rates on the US error reporting API

Delays with notifications

Honeybadger Services, Components and Regions

Recent Honeybadger events

Elevated failure rates on the US error reporting API

Elevated error rates on the US error reporting API

Delays with notifications

Outage in check-in processing

Insights queries and dashboards are unavailable

Error ingestion API is down