Mar <var data-var='date'>12</var>, <var data-var='time'>12:27</var> PDT Resolved - We had a backlog in our notifications queue that prevented timely delivery of alerts to channels such as Email, Slack, etc. The backlog has been processed as of 19:20 UTC. Our autoscaling rules for this service had a monitoring gap that prevented a scale-out to maintain a low latency on the queue. This monitoring gap has been resolved.

Delays with notifications

Jan <var data-var='date'>14</var>, <var data-var='time'>12:48</var> PST Resolved - This incident has been resolved.Jan <var data-var='date'>14</var>, <var data-var='time'>12:40</var> PST Identified - A recent deploy has resulted in our rejecting some check-in API requests. We are in the process of deploying a fix.

Outage in check-in processing

Oct <var data-var='date'>22</var>, <var data-var='time'>13:15</var> PDT Resolved - This incident has been resolved.Oct <var data-var='date'>22</var>, <var data-var='time'>12:46</var> PDT Identified - The issue has been identified and a fix is being implemented.

Insights queries and dashboards are unavailable

Oct <var data-var='date'>22</var>, <var data-var='time'>06:16</var> PDT Resolved - The EU API has been restored.Oct <var data-var='date'>22</var>, <var data-var='time'>06:11</var> PDT Monitoring - The US API has been restored, and the EU API restoration is in process.Oct <var data-var='date'>22</var>, <var data-var='time'>06:09</var> PDT Identified - API ingestion of errors has been interrupted. We are deploying a fix.

Error ingestion API is down

Oct <var data-var='date'>20</var>, <var data-var='time'>12:58</var> PDT Resolved - This incident has been resolved.Oct <var data-var='date'>20</var>, <var data-var='time'>12:58</var> PDT Monitoring - Processing has been restored and all systems are normal. We'll continue to monitor as AWS continues to restore service. Thanks!Oct <var data-var='date'>20</var>, <var data-var='time'>10:55</var> PDT Update - We're continuing to monitor the AWS outage. Errors are being processed, but may be delayed. Uptime checks are also delayed, so you may notice lag with uptime notifications and status page updates.Oct <var data-var='date'>20</var>, <var data-var='time'>06:44</var> PDT Identified - Our error processing pipeline has been impacted by the AWS outage earlier this morning. Errors are being processed, but the end-to-end latency is much higher than normal. We are working to restore the processing to full capacity.

Delays in error processing

Oct <var data-var='date'>20</var>, <var data-var='time'>01:00</var> PDT Resolved - Our systems were affected by the DynamoDB outage in AWS' us-east-1 region. Impacts included not being able to log use the web app, uptime checks not being executed on time, and check-ins being falsely reported as missing. We are evaluating what steps we can take to mitigate these kinds of impacts from this kind of outage in the future. Additionally, our status page was not updated in a timely manner due to our not having configured various AWS services as dependent components of our status page. We have fixed that issue, so future AWS outages will result in our status page automatically being updated.

AWS outage in us-east-1

Feb <var data-var='date'>12</var>, <var data-var='time'>05:30</var> PST Resolved - We had an interruption in processing of check-in alerts and in-app stats from about 12:30 to about 13:30 UTC. This may have resulted in a failure or a delay in reporting check-ins that went missing during that period.

Delay in stats and check-in processing

Dec <var data-var='date'>20</var>, <var data-var='time'>06:05</var> PST Completed - The scheduled maintenance has been completed. Some check-ins may have falsely reported as being missing during this window. We will work on improving that for future maintenance events.Dec <var data-var='date'>20</var>, <var data-var='time'>06:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Dec <var data-var='date'>19</var>, <var data-var='time'>14:31</var> PST Scheduled - We will have a few minutes of app unavailability and error ingestion delays during a database upgrade.

Database maintenance

Oct <var data-var='date'>22</var>, <var data-var='time'>08:30</var> PDT Resolved - The backlog has been processed and all systems are running normally.Oct <var data-var='date'>22</var>, <var data-var='time'>06:35</var> PDT Monitoring - We have identified and corrected an issue in our data processing pipeline that may have caused delays for some customers. No data has been lost and the system should be caught up shortly.

Data Processing Delays

Sep <var data-var='date'>23</var>, <var data-var='time'>04:57</var> PDT Resolved - This incident has been resolved.Sep <var data-var='date'>23</var>, <var data-var='time'>04:48</var> PDT Monitoring - We have identified the cause of the slowdown and performance is back to normal.Sep <var data-var='date'>23</var>, <var data-var='time'>03:37</var> PDT Investigating - We are looking into the cause of slowness and timeouts with the web app.

App slowness and timeouts

Jul <var data-var='date'>24</var>, <var data-var='time'>20:06</var> PDT Resolved - The backlog has been fully processed, so all Insights events should be ingested and available.Jul <var data-var='date'>24</var>, <var data-var='time'>18:55</var> PDT Monitoring - We have successfully pushed a fix and are waiting for the backlog to be processed. I also want to give a reminder that this only affects Insights events; error processing was unaffected.Jul <var data-var='date'>24</var>, <var data-var='time'>16:42</var> PDT Identified - We have identified an issue with Insights event ingestion backlog. We are working on a fix.

Insights backlog issue

Feb <var data-var='date'>12</var>, <var data-var='time'>12:39</var> PST Resolved - We discovered and addressed a configuration issue that caused the delay in processing. Processing has returned to normal.Feb <var data-var='date'>12</var>, <var data-var='time'>12:22</var> PST Investigating - We are seeing an increased backlog in our check-in processing, which is causing false alarms to be reported. We are investigating the cause of the backlog.

Check-in monitoring

Feb <var data-var='date'> 6</var>, <var data-var='time'>15:19</var> PST Resolved - This incident has been resolved.Feb <var data-var='date'> 6</var>, <var data-var='time'>15:03</var> PST Monitoring - We've resolved an issue with query contention, and web application service has been restored. Our API pipeline was unaffected by this incident. Some charts on the reports tab may be outdated until we update project counts. We'll continue to monitor the situation. Sorry for the inconvenience!Feb <var data-var='date'> 6</var>, <var data-var='time'>13:42</var> PST Identified - We've identified an issue with aggregate database queries and are working on a fix.Feb <var data-var='date'> 6</var>, <var data-var='time'>13:02</var> PST Update - We are continuing to investigate this issue.Feb <var data-var='date'> 6</var>, <var data-var='time'>12:59</var> PST Investigating - We are currently experiencing elevated response times and timeouts from our web application.

Increased response times and timeouts

Jan <var data-var='date'>17</var>, <var data-var='time'>11:21</var> PST Resolved - This incident has been resolved.Jan <var data-var='date'>17</var>, <var data-var='time'>11:19</var> PST Identified - We are working on restoring service

Web app is unavailable

Nov <var data-var='date'>10</var>, <var data-var='time'>11:44</var> PST Resolved - We discovered this morning that our Logplex pipeline, which handles Heroku platform errors, had an interruption in processing, which caused platform errors not to be recorded. This has been corrected, the backlog is being processed, and additional monitoring has been added to avoid this issue in the future.

Delay in Logplex processing

Jun <var data-var='date'>13</var>, <var data-var='time'>14:37</var> PDT Resolved - Lambda is back in business, and the backlog of events triggered during the outage is being processed.Jun <var data-var='date'>13</var>, <var data-var='time'>12:34</var> PDT Update - Search indexing and notice timeline charts are also impacted by the failures with Lamba. We're continuing to queue updates for processing once Lambda resumes normal operations.Jun <var data-var='date'>13</var>, <var data-var='time'>11:59</var> PDT Identified - Issues with AWS Lambda are causing our uptime check reports to be delayed. Reports are queued up and will be delivered as AWS services recover.

Uptime check reports are delayed

Apr <var data-var='date'> 8</var>, <var data-var='time'>07:30</var> PDT Completed - The scheduled maintenance has been completed.Apr <var data-var='date'> 8</var>, <var data-var='time'>07:01</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Apr <var data-var='date'> 6</var>, <var data-var='time'>14:45</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Mar <var data-var='date'>24</var>, <var data-var='time'>15:30</var> PDT Resolved - Starting around 3:30pm PST, we had an incident where an unhealthy API server was in rotation in our load balancer. During that time requests routed to that server responded with a 502 which could have resulted in the clients dropping notices. At around 5:30pm PST we removed the unhealthy server from rotation. 7:22pm PST edit: Reworded message to describe that notices are not queued on the client, but dropped when the server responds in an error state

Unhealthy API server

Mar <var data-var='date'>24</var>, <var data-var='time'>02:01</var> PDT Resolved - We identified that our workers had an issue deploying. We deployed a fix and upped our capacity to deal with the current backlog. Everything is back to normal now. Turns out this issue was related GitHub rotating their public RSA key: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/Mar <var data-var='date'>24</var>, <var data-var='time'>01:40</var> PDT Identified - We have identified the issue and are working towards clearing the backlog.Mar <var data-var='date'>24</var>, <var data-var='time'>01:05</var> PDT Investigating - We currently are having issues processing our notice backlog. We are looking into the issue.

Issues processing backlog

Mar <var data-var='date'> 9</var>, <var data-var='time'>01:40</var> PST Resolved - We had an incident (from 1:00 to 1:27 PST) where our apps were unresponsive. They have since recovered.

Web app unavailable

Feb <var data-var='date'>14</var>, <var data-var='time'>08:22</var> PST Resolved - Our primary redis instance was having a bad time. This has been fixed, and now we're back to normal.Feb <var data-var='date'>14</var>, <var data-var='time'>08:10</var> PST Investigating - We are seeing increased error rates for our API

API issues

Dec <var data-var='date'>16</var>, <var data-var='time'>15:07</var> PST Resolved - This incident has been resolved.Dec <var data-var='date'>16</var>, <var data-var='time'>14:21</var> PST Update - We are continuing to monitor for any further issues.Dec <var data-var='date'>16</var>, <var data-var='time'>11:19</var> PST Monitoring - Our email provider is having problems delivering emails, so our outbound email deliveries are being delayed.

Email delivery is being delayed

Nov <var data-var='date'>23</var>, <var data-var='time'>01:23</var> PST Resolved - This incident has been resolved.Nov <var data-var='date'>23</var>, <var data-var='time'>01:17</var> PST Investigating - We are currently investigating this issue.

Aug <var data-var='date'> 2</var>, <var data-var='time'>18:19</var> PDT Resolved - This incident has been resolved.Aug <var data-var='date'> 2</var>, <var data-var='time'>18:08</var> PDT Investigating - We are currently investigating this issue.

Uptime checks are delayed

Jul <var data-var='date'>13</var>, <var data-var='time'>09:15</var> PDT Resolved - We had a network disruption that lasted approximately one minute. The cause has been identified and resolved.

Is Honeybadger down?

Honeybadger status is UP

Lastest published events

Delays with notifications

Outage in check-in processing

Insights queries and dashboards are unavailable

Honeybadger Services, Components and Regions

Recent Honeybadger events

Delays with notifications

Outage in check-in processing

Insights queries and dashboards are unavailable

Error ingestion API is down

Delays in error processing

AWS outage in us-east-1

Get your FREE account now,
5 monitors included!

Is Honeybadger down?

Honeybadger status is UP

Lastest published events

Delays with notifications

Outage in check-in processing

Insights queries and dashboards are unavailable

Honeybadger Services, Components and Regions

Recent Honeybadger events

Delays with notifications

Outage in check-in processing

Insights queries and dashboards are unavailable

Error ingestion API is down

Delays in error processing

AWS outage in us-east-1

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!