Sep <var data-var='date'>30</var>, <var data-var='time'>13:11</var> PDT Resolved - This incident has been resolved. Sep <var data-var='date'>30</var>, <var data-var='time'>09:25</var> PDT Update - The root cause of issues with large repos has been fully addressed. We are continuing to clear some backlogged background jobs. We will continue to monitor and report updates here. Sep <var data-var='date'>30</var>, <var data-var='time'>06:19</var> PDT Monitoring - Users in Europe and US east coast with large repos (5k+ repos) experienced slow build completion, starting around 2am PST / 5am EST. Behavior: PRs opened between 2am-6am PST may have slow or missing Coveralls PR comments or status checks. We have identified the issue and allocated resources to clear a backlog of background jobs. Jobs should clear and builds should complete in the next ~1hr.

Increased latency for Large Repos (5k+ source files)

Sep <var data-var='date'>28</var>, <var data-var='time'>10:21</var> PDT Resolved - This incident has been resolved. Sep <var data-var='date'>28</var>, <var data-var='time'>10:02</var> PDT Update - We are continuing to monitor for any further issues. Sep <var data-var='date'>28</var>, <var data-var='time'>09:14</var> PDT Update - Operational in all regions. We are continuing to monitor for any further issues. Sep <var data-var='date'>28</var>, <var data-var='time'>09:13</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'>28</var>, <var data-var='time'>09:11</var> PDT Identified - The issue has been identified and a fix is being implemented. Sep <var data-var='date'>28</var>, <var data-var='time'>09:03</var> PDT Investigating - We are currently investigating this issue.

Elevated errors rates

Aug <var data-var='date'>11</var>, <var data-var='time'>14:09</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'>11</var>, <var data-var='time'>12:55</var> PDT Monitoring - This was the second wave of the attack we experienced this weekend, broadened and tripled in volume. We implemented a second phase of defense which appear to be working and will continue monitoring for the next hour. Aug <var data-var='date'>11</var>, <var data-var='time'>12:02</var> PDT Investigating - We are currently experiencing occasional "Website under heavy load" warnings. We believe this is the second wave of an attack we experienced this weekend involving an army of scraper bots which we successfully mitigated. We believe the attacker may have gotten around an initial phase of defense. Separate from a DDoS attack, these are millions of individual requests coming from over 280K different IP addresses across 12 different countries, so our standard DDoS protection (IP quarantine) does not apply. We are implementing a second phase of defense and will report updates here.

"Website under heavy load" warning

Jul <var data-var='date'>20</var>, <var data-var='time'>10:32</var> PDT Resolved - This issue was resolved over the weekend. We will continue to monitor for elevated latency in the modified queues. Jul <var data-var='date'>17</var>, <var data-var='time'>08:48</var> PDT Update - Latency remains elevated by about 30% for large repos. We are continuing to monitor and do what we can to mitigate until we can implement a longer-term solution. Jul <var data-var='date'>16</var>, <var data-var='time'>11:25</var> PDT Update - We are continuing to proactively monitor and clear the large repo queues. We are working on a longer term solution over the weekend and will post about it in the postmortem for this issue when complete. We will keep this incident open as long as we have large repo queues with above-average latency. Jul <var data-var='date'>15</var>, <var data-var='time'>17:47</var> PDT Update - We are continuing to monitor for any further issues. Jul <var data-var='date'>15</var>, <var data-var='time'>11:14</var> PDT Update - We have reduced traffic in affected queues by 75%. We still expect to clear the queue within about 1 hr. We will post updates here. If you think you may have been paused as a user with a large repo (>=5K source files) that has uploaded 3-10x normal daily uploads, please reach out to us at support@coveralls.io and we'll confirm and give you some next steps to avoid further pauses (see Usage Add-Ons at https://coveralls.io/pricing). Jul <var data-var='date'>15</var>, <var data-var='time'>10:28</var> PDT Monitoring - All systems fully functional. We have identified increased latency for large repos again and have proactively arrested sources of excess requests. We expect to restore normal latency within the hour.

Increased latency for large repos

Jul <var data-var='date'>13</var>, <var data-var='time'>07:56</var> PDT Resolved - This incident has been resolved but we believe it triggered a worsened incident overnight Sun night/Mon morning (US PDT) which has just been resolved. To be confirmed by full RCA, we believe a deluge of large repo uploads tied up individual web servers that handle frontline requests. Each web server is able to recover on its own, and did, but as volume increased all servers were eventually affected, only allowing short windows where requests could get through—rejecting most requests with 504 errors. We'll post a post-mortem when we understand more about what happened and how to prevent it going forward. Jul <var data-var='date'>10</var>, <var data-var='time'>10:34</var> PDT Monitoring - We are continuing to monitor this situation. Latency is much reduced but still elevated. We will close this incident when it's fully restored to normal. Jul <var data-var='date'>10</var>, <var data-var='time'>08:00</var> PDT Identified - We are monitoring increased latency for larger repos (5K+ source files) due to elevated traffic in background job queues. We will pause outlier repos (more than 2x average traffic) to allow queues to clear for the general population, then restore once queues have cleared. If you think your repo may have been one of those paused, two things: 1) Reach out to us at support@coveralls.io and we'll confirm; and 2) Consider purchasing a usage add-on so that your jobs are not throttled for fair-use, or an enterprise plan with isolated infrastructure for maximum performance: https://coveralls.io/pricing

Jul <var data-var='date'> 7</var>, <var data-var='time'>21:39</var> PDT Resolved - Increased latency for large repos has been resolved for the general public. We will be clearing three outlier repos overnight, which should be fully cleared by tomorrow AM. Jul <var data-var='date'> 7</var>, <var data-var='time'>15:18</var> PDT Monitoring - We've had too offload some outlier repos to let the general public of users with heavy repos to clear. There were three such repos. We will restart processing on those repos as soon as general queues have cleared. If you think you may be one of the repos with excessive traffic that was paused today, just reach out and we'll let you know: support@coveralls.io. We expect to clear the general queues by 6pm PDT. After that we will restore processing for outlier and expect that to complete by this evening between 10p-12a PDT. Jul <var data-var='date'> 7</var>, <var data-var='time'>09:43</var> PDT Investigating - We are seeing increased latency for larger repos (5K source files and greater). While the site is fully operational, you may experience increased delays before receiving finished build pages and coverage notifications to your PRs if your repo had >=5K source files. We are monitoring and will pause any outlier repos (repos with unexplained, excessive usage) to allow general traffic in large repo queues to clear.

May <var data-var='date'>28</var>, <var data-var='time'>08:53</var> PDT Resolved - We are closing this incident. Our main background processing queue for larger repos has had no backups in 48 hrs. We will continue monitoring for recurrence. May <var data-var='date'>27</var>, <var data-var='time'>15:18</var> PDT Update - We've cleared the backlog of background jobs for larger projects from our heavy project queues, with the exception of nine projects whose volume continues to exceed our daily fair use thresholds. Those nine have been redirected to secondary and tertiary queues that drain only after the primary queue is clear. Most of their jobs have now been processed, but a few projects generate more jobs than a normal day + overnight cycle can drain, so some backlog remains. We're continuing to work those queues down manually. If you think one of your projects may be affected, reach out to us at support@coveralls.io and we'll check status for you. May <var data-var='date'>25</var>, <var data-var='time'>13:06</var> PDT Update - We are continuing to monitor for any further issues. May <var data-var='date'>21</var>, <var data-var='time'>09:23</var> PDT Update - We continue to monitor background processing queues for larger repos (5K+ files), as we continue offloading incoming jobs from the outlier repos whose volume led to the original backlog. If you think you may have been paused as one of these repos with outlier-level volume, please reach out to us at support@coveralls.io and we'll confirm status and get you an ETA on when your remaining jobs will be processed. May <var data-var='date'>20</var>, <var data-var='time'>07:06</var> PDT Update - We have moved all repos with more than 300 jobs in queue to dedicated queues for processing. These 7 repos were responsible for 75% traffic in our heavy repos queue. This move allows us to provide normal processing times for non-outlier repos and dedicated resources to outlier repos. Outlier repos by their job volume will still take longer to clear. If you believe you may be one of these outlier repos, please reach out and we'll confirm and give you an idea of when your jobs will be processed. If you'd like faster processing, we can also pause processing on older job, for instance, anything older than 1 hour (or 2 hours, or 3 hours) ago. May <var data-var='date'>20</var>, <var data-var='time'>06:48</var> PDT Update - We are keeping this incident open as we continue monitoring. May <var data-var='date'>19</var>, <var data-var='time'>06:36</var> PDT Update - We are still receiving reports of latency for large projects and are seeing backlogs of background jobs in queues dedicated to large repos (5K+ source files). We may pause some outlier repos with excessive numbers of jobs in order to clear general traffic. If you believe you may have been paused, reach out and we'll let you know. May <var data-var='date'>18</var>, <var data-var='time'>10:37</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>18</var>, <var data-var='time'>09:41</var> PDT Identified - We have received reports of increased report latency for larger projects (5K file and up). We are doing our best to clear the background job queues for these projects. If you have one of these projects, feel free to reach out and we'll check status for you.

Increased latency for large projects

May <var data-var='date'>10</var>, <var data-var='time'>08:11</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>10</var>, <var data-var='time'>07:44</var> PDT Investigating - We are currently investigating an issue with an apparently expired SSL cert. Our orchestration layer shows no issue with certs so we need to dig deeper.

Unschedule maintenance

Feb <var data-var='date'>27</var>, <var data-var='time'>14:30</var> PST Resolved - This incident has been resolved. Feb <var data-var='date'>27</var>, <var data-var='time'>12:23</var> PST Update - After 1-hour of monitoring, we are cautiously optimistic that service is fully restored in all regions. Feel free to confirm service in your area (with thanks): ops@coveralls.io And please let us know if you are still having any issues: support@coveralls.io We will continue to monitor throughout the rest of the day, and weekend. Finally, please come back for our post-mortem. We are aiming for the first part of next week, but our priority is making sure service is stable. For all of the kindness and support, we thank you. To everyone, thanks for your patience. We'll be in touch. Feb <var data-var='date'>27</var>, <var data-var='time'>11:19</var> PST Update - We would love to hear from anyone outside of the US as to whether service has been restored for you: ops@coveralls.io We have no reason to believe service hasn't been restored to all regions, but we want to make sure. We will continue monitoring. Feb <var data-var='date'>27</var>, <var data-var='time'>11:11</var> PST Update - We are back, for at least some users. We’ll stay at “partial outage” until we confirm full restoration. Feb <var data-var='date'>27</var>, <var data-var='time'>11:02</var> PST Update - We are seeing systems coming back online. We assume service has been restored at some level. We will assess and update you here ASAP. Feb <var data-var='date'>27</var>, <var data-var='time'>07:41</var> PST Update - Day 4 | Morning Status: Unresolved — No ETA It's the morning of Day 3, and service has still not been restored. We want you to know we're here, fully aware, and have not stopped working on this. We remain in constant communication with our hosting provider's account team, who we have every reason to believe is continuing to advocate for us internally. But the reality remains the same: we do not have an ETA, and we do not have direct control over the outcome we need: restoration of our services and access to our infrastructure. As we pass milestones in this outage that we never imagined we would see (and haven't in 15 years of operation) the path forward is becoming clearer. We need to find a new provider, at minimum for failover, and potentially as our primary host. We've been reading the tweets, watching the YouTubes, and evaluating options. But if anyone in a position similar to ours has migrated away from a primary tier host and successfully manages a write-intensive service with hundreds of thousands of daily uploads, we would genuinely love to hear from you about your experience. Please reach out: ops@coveralls.io Because if our services are not restored by the end of today, we will be standing up new infrastructure over the weekend. We will continue to post updates as we have them. Feb <var data-var='date'>26</var>, <var data-var='time'>17:13</var> PST Update - Day 3 | Evening Status: Unresolved — No ETA We want to be transparent with you: after two full days of working directly with our hosting provider’s account team, we still do not have an ETA for service restoration. Despite having internal sponsorship within our provider’s organization, we have been unable to get the traction needed to resolve what appears to have been an automated action, one that no one on their side has taken ownership of causing or fixing. We understand the severity of this disruption. Your CI workflows depend on Coveralls, and we can see the impact this outage is having on your teams. That is not lost on us for a single moment. This experience has made something painfully clear: no matter how trusted or established a cloud provider may be, a single vendor dependency without a failover plan is an unacceptable risk, for us and for you. We are already evaluating backup infrastructure options to ensure that nothing like this can take our service offline this way again. We’ll share more in our full postmortem. In the meantime, to those of you who have stayed patient and stuck with us through this, thank you. We see you, we appreciate you, and we are committed to making this right. If there is anything we can do to help in the interim, please reach out. We will continue to post updates as we have them. Feb <var data-var='date'>26</var>, <var data-var='time'>06:53</var> PST Update - As of 6am PST Thu, Feb 26, we are still waiting on restoration. We thought this would be resolved by this morning. We are in touch with our account team and are doing what we can to restore service. We apologize for the ongoing inconvenience. Feb <var data-var='date'>25</var>, <var data-var='time'>17:56</var> PST Update - After a full day of working with our hosting provider's account team (who've been really helpful), we've discovered that this outage stems from an account restriction that, by all indications, shouldn't have happened (we believe the technical term is "snafu"). They are actively working to get it reversed. This is not a technical failure — our infrastructure and all customer data are fully intact. Services will be restored as soon as account access is. We're hopeful for resolution this evening or tomorrow morning. We'll have more to say once we're back online. Feb <var data-var='date'>25</var>, <var data-var='time'>09:49</var> PST Update - We are continuing to communicate with our hosting provider around any updates or ETA. Feb <var data-var='date'>25</var>, <var data-var='time'>09:48</var> PST Monitoring - We are experiencing an outage that appears to have originated at our infrastructure provider. We are unable to access our account and are working with our account manager to resolve the issue. Unfortunately, due to the nature of the incident, we have no way to establish an ETA. We are doing everything we can to restore service, and will update our status page as soon as we have a fix or an ETA. Please follow along here: https://status.coveralls.io/ You can subscribe for email updates by clicking SUBSCRIBE TO UPDATES button in the top right corner of this page. Feb <var data-var='date'>24</var>, <var data-var='time'>19:28</var> PST Update - Outage will be extended for at least several more hours. To avoid further disruption to your CI workflows, we recommend employing the fail-on-error: false input option available with all official coveralls integrations. See: https://docs.coveralls.io/integrations#official-integrations. Official Integration: fail-on-error option - GitHub Action: `fail-on-error: false` - CircleCI Orb: `fail_on_error: false` - Coverage Reporter (CLI): `--no-fail` Feb <var data-var='date'>24</var>, <var data-var='time'>17:36</var> PST Identified - The issue has been identified and a fix is being implemented. Feb <var data-var='date'>24</var>, <var data-var='time'>14:58</var> PST Update - We are continuing to investigate this issue. Feb <var data-var='date'>24</var>, <var data-var='time'>14:52</var> PST Investigating - We are currently investigating this issue.

Service outage (RESTORED, MONITORING)

Feb <var data-var='date'> 2</var>, <var data-var='time'>09:33</var> PST Resolved - Just a note to address the gap in our Coverage Calculation Background Job Dequeue Time Graph today, MON, FEB 2 from 00:15:00 PST to 07:35:00 PST: Coveralls experienced no disruption in service at this time. Instead, a deployment issue cause the cron job that reports the metric to fail until it was resolved at 07:35:00 PST.

All Systems Operational

Jan <var data-var='date'>10</var>, <var data-var='time'>23:00</var> PST Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>10</var>, <var data-var='time'>19:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>10</var>, <var data-var='time'>18:02</var> PST Scheduled - We will be undergoing scheduled maintenance during this time.

Infrastructure Maintenance

Dec <var data-var='date'>19</var>, <var data-var='time'>07:24</var> PST Resolved - We had to pause some queues to recover performance for new jobs, but will clear those as soon as we've recovered normal build times for new jobs (ETA: 15-min). If you are a user in APAC or EU, you may have had your jobs paused. One repo in particular has represented 90% overnight workload. We will reach out to that user. Dec <var data-var='date'>19</var>, <var data-var='time'>06:52</var> PST Monitoring - We are continuing to monitor as we scale to clear backlogged jobs. Dec <var data-var='date'>19</var>, <var data-var='time'>06:34</var> PST Identified - We are continuing to experience elevated latency in APAC and EU.

Elevated Latency in APAC and EU

Dec <var data-var='date'>18</var>, <var data-var='time'>08:58</var> PST Resolved - Build times are normal for all new builds. Background queues have been cleared of 99% of backlogged jobs; the only ones that remain are for large repos (>5K source files), which should clear in the next 30-45 minutes depending on size. No further effects on latency expected. Closing this incident. Dec <var data-var='date'>18</var>, <var data-var='time'>07:31</var> PST Update - Update: - Normal build times have been recovered for all new jobs. - Backlogged jobs are 80% cleared (20% remaining). We will continue clearing backlogged jobs until cleared and monitoring for any further issues. Dec <var data-var='date'>18</var>, <var data-var='time'>06:51</var> PST Monitoring - We had an incident overnight that entailed several server outages affecting APAC and EU users. The impact has been latency in recent build times for those users. We have resolved the issue and are scaling to clear backlogged background jobs. US Central and West users should not be affected.

Nov <var data-var='date'>17</var>, <var data-var='time'>11:43</var> PST Resolved - This incident has been resolved. Build times for all new builds is normal across the board. We are still clearing a backlog of background jobs, which should be clear in the next 15-20-min. If you are having issues with a slow or stuck build, feel free to reach out to us at support@coveralls.io. These steps will save time: 1) Mention this incident: https://status.coveralls.io/incidents/hkqt790213m5 2) Share your Coveralls Build URL (from your CI build log), or your CI build number. Nov <var data-var='date'>17</var>, <var data-var='time'>09:28</var> PST Update - Build times are returning to normal for all builds created since 05:00AM PDT. Older builds are being processed by scaled resources and should clear in 45min to 1 hr. If you are having issues with a slow or stuck build, especially if priority, feel free to reach out to us at support@coveralls.io. These steps will save time: 1) Mention this incident: https://status.coveralls.io/incidents/hkqt790213m5 2) Share your Coveralls Build URL (from your CI build log), or your CI build number. Nov <var data-var='date'>17</var>, <var data-var='time'>08:35</var> PST Monitoring - We received reports from EU customers of elevated latency and have resolved the cause. We will be managing workloads to ensure standard processing times for all new builds, and scaling resources to clear delayed builds. Follow this incident for updates.

Elevated Latency

Nov <var data-var='date'>10</var>, <var data-var='time'>10:55</var> PST Resolved - We have attempted to improve latency this morning for new jobs from users in EU and US time zones, which has meant offloading older jobs to specialized queues with scaled up resources, which, at this point, have already drained. System-wide, latency continues improving for all users and should reach normal in 15-30 minutes for most users. If you are still experiencing elevated latency after that, or have any jobs from CI builds run in the last 24 hrs that have not completed, please reach out to us at support@coveralls.io and we'll investigate to determine whether your builds were caught up in this incident, or if they have a different cause. NOTE: Missing data points in the "DEQUEUE" Graph on our main Status Page does not indicate that processing stopped during that period, just that the server that reports our stats lost comms during that period. As stated elsewhere, to avoid this confusion in the future it's one of our goals to spread stats collection across all servers. Nov <var data-var='date'>10</var>, <var data-var='time'>08:31</var> PST Update - We have confirmed elevated latency for global users in ASIA, AU and EU starting on Mon, Nov 10 around 09:15a UTC (01:15a PDT). We have resolved the cause but will be performing RCA in the next 24-48 hrs to understand the triggering conditions and avoid further incidents. In the meantime, we will be monitoring latency and will likely manage workloads to improve latency for new jobs (from currently active time zones), routing jobs from inactive time zones to new resources to clear the backlog. We will upgrade performance from Degraded to (Fully) Operational when complete. Nov <var data-var='date'>10</var>, <var data-var='time'>08:23</var> PST Identified - The issue has been identified and a fix is being implemented. Nov <var data-var='date'>10</var>, <var data-var='time'>08:23</var> PST Update - We are continuing to investigate this issue. Nov <var data-var='date'>10</var>, <var data-var='time'>08:14</var> PST Investigating - We are currently investigating this issue.

Oct <var data-var='date'>24</var>, <var data-var='time'>14:54</var> PDT Resolved - We are closing this incident having received no further reports of 504 errors today. We will continue to monitor for them. Oct <var data-var='date'>24</var>, <var data-var='time'>11:35</var> PDT Monitoring - We received reports of 504 Gateway Timeouts from our Web Servers (coverage report uploads) between 6:00am-6:20am PDT and several moments ago between 11:15am-11:30am PDT. We have recovered service to all web servers and are monitoring for further occurrences.

504 Gateway Timeouts (Resolved)

Oct <var data-var='date'>13</var>, <var data-var='time'>16:15</var> PDT Resolved - We have received no further reports of 504 timeout errors on coverage report uploads today, but we continue to monitor and will continue trying to improve our mitigations. Oct <var data-var='date'>13</var>, <var data-var='time'>11:02</var> PDT Monitoring - We have received some reports of 504 errors this morning. We are monitoring to avoid further occurrences. We are looking into any gaps in our new monitoring regime, which has drastically reduced occurrences, to see where we can further improve.

Some reports of 504 Timeouts

Oct <var data-var='date'>10</var>, <var data-var='time'>12:00</var> PDT Resolved - This incident is resolved. We have applied an additional layer of monitoring that should help us catch these cases earlier. Oct <var data-var='date'>10</var>, <var data-var='time'>09:46</var> PDT Update - Based on the details of today's incident, we've added another layer of monitoring that should give us earlier indications of this error state and allow us to respond more quickly. Oct <var data-var='date'>10</var>, <var data-var='time'>08:23</var> PDT Monitoring - We've received results of 504 Timeout errors from 7am PDT through 8:15am PDT. We have addressed the issue and issue was resolved as 8:19am PDT. We are monitoring for further issues.

504 Timeouts (Resolved)

Sep <var data-var='date'>29</var>, <var data-var='time'>08:56</var> PDT Resolved - We are closing this incident after recent mitigations and a weekend without any reports. We continue to implement mitigations and infrastructure changes we believe will further reduce incidents of this error type. Sep <var data-var='date'>26</var>, <var data-var='time'>08:28</var> PDT Monitoring - All systems operational. During our US overnight hours (PDT / GMT-7), we received additional reports of 504 timeout errors affecting coverage uploads. The issue has since been resolved, and we are actively monitoring for further occurrences. We have already implemented several mitigations over the past few weeks, and additional measures will be deployed today to further reduce the likelihood of these errors, particularly for our international customers. If you experience a 504 error, the most helpful details you can provide are: 1. A timestamp or timeframe of the error 2. Any Cloudflare Ray ID shown on the error page Please share these details in our public tracking issue here: https://github.com/lemurheavy/coveralls-public/issues/1824 These reports are invaluable as we continue to investigate and refine our mitigations.

504 Timeout Errors on Coverage Uploads

Sep <var data-var='date'>24</var>, <var data-var='time'>17:53</var> PDT Resolved - 500 Internal Server Errors on Uploads The recent 500 error surfacing during some coverage uploads as: > ⚠️ Internal server error. Please contact Coveralls team. has been resolved. A full postmortem will be published here soon. In the meantime, you can find more detail in the main tracking issue: https://github.com/coverallsapp/coverage-reporter/issues/180 Summary The root cause was ultimately infrastructure-related, not a regression in recent coverage-reporter releases. The previous workaround of pinning your coverage-reporter version is therefore not required. We have decided to close this incident, which we intentionally kept open for over a week to track a series of 504 and 5xx issues with overlapping root causes. In hindsight, the broadened scope made updates less clear than we'd hoped. With today’s resolution and the mitigations applied throughout the week, the occurrence of 504 errors during uploads (POSTs) has been significantly reduced. Going forward, any new 504 errors should be considered unexpected, isolated events. At the same time, we continue work on several instances of intermittent GET-related 504 errors affecting: - Source File pages - Repo pages - Add Repos pages Progress on those issues will be reported separately here: https://github.com/lemurheavy/coveralls-public/issues/1757 Sep <var data-var='date'>23</var>, <var data-var='time'>12:14</var> PDT Update - Fix for unrelated 500 errors: If you receive a `500` error with this error message format: > ⚠️ Internal server error. Please contact Coveralls team. Please know it is unrelated to the `504` errors being monitored in this open incident. Those, intermittent `500` errors are caused by a regression in one of the latest coverage-reporter releases: `v0.6.16` or `v0.6.17`. Workaround: Pin your coverage-reporter-version to `v0.6.15` in your integration config. For thorough instructions, see this public issue: https://github.com/coverallsapp/coverage-reporter/issues/180 We’re investigating the root cause and will post updates once a fix is released. Sep <var data-var='date'>22</var>, <var data-var='time'>09:43</var> PDT Update - Mitigated – Monitoring All systems operational. Recent mitigations, including fleet expansion and autoscaling, have reduced 504 timeout reports significantly. The remaining reports are infrequent and occur mostly during overnight and weekend hours (PDT). We are continuing to monitor closely and are working on a multi-part solution to eliminate all known causes. Until then, we are keeping this incident open in Monitoring. We will close it once 504 errors have returned to being unexpected, isolated events. Sep <var data-var='date'>15</var>, <var data-var='time'>11:57</var> PDT Update - Mitigation in place. All systems operational. This morning we deployed additional capacity and autoscaling measures to reduce 504 errors on coverage report uploads: - Doubled our web server fleet (on top of the prior doubling when this issue began). - Enabled autoscaling at the web layer, allowing the fleet to double again automatically when NGINX response times exceed thresholds. The underlying trigger remains rare surges of upload requests from outlier repositories (750–1250 uploads per build). While we have paused processing for these repos, our HTTP servers must still handle the incoming requests until they stop. Timezone coverage: As a small team based in Los Angeles (PDT), our ability to respond in real time is most limited overnight (10p–6a PDT). Unfortunately, the primary outlier repos are in APAC, making this the window of highest risk. With these changes, we hope to reduce the occurrence of upload 504s during this window. We will monitor results closely and continue tuning autoscaling thresholds. Please let us know if you continue to see 504 errors on uploads. Sep <var data-var='date'>12</var>, <var data-var='time'>08:44</var> PDT Update - All systems operational. Earlier today (6:45–7:45 AM PDT), we received elevated reports of 504 timeout errors. We have not been able to reproduce the issue since, but if you are still experiencing errors, please contact us at support@coveralls.io. The affected areas may include: - Coverage Report Uploads (/api/v1/jobs) - Add Repos Page - Repo Page - Source File Page Fixes for the Add Repos, Repo, and Source File pages are scheduled to be deployed by end of day (PDT). Sep <var data-var='date'> 9</var>, <var data-var='time'>08:24</var> PDT Update - All systems operational. We have released one of two parts of a near-term solution into production resolving a minority subset of 504 errors. We are still working on releasing part two into production. Subscribe for updates at this status page, or follow this public tracking issue for updates: https://github.com/lemurheavy/coveralls-public/issues/1757 Sep <var data-var='date'> 8</var>, <var data-var='time'>11:57</var> PDT Update - All systems operational. Continuing to keep this open until we have released our short-term fix into production. Subscribe for updates at this status page, or follow this public tracking issue for updates: https://github.com/lemurheavy/coveralls-public/issues/1757 Sep <var data-var='date'> 5</var>, <var data-var='time'>09:05</var> PDT Update - We are still working on a near-term fix. We will post here, and here when complete: https://github.com/lemurheavy/coveralls-public/issues/1757 Sep <var data-var='date'> 3</var>, <var data-var='time'>11:00</var> PDT Monitoring - We’re currently seeing elevated reports of 504 Timeout errors affecting some customers on a subset of Coveralls pages, including: - Source File pages - Repo pages - Add Repos pages All systems and pages are generally operational; a subset of customers are experiencing these errors, sometimes intermittently. There is a public tracking issue for the Source File timeout errors here: https://github.com/lemurheavy/coveralls-public/issues/1757 Fix in progress: We’re implementing a short-term fix over the next 24–48 hours, which should eliminate the timeouts. A longer-term fix is also planned, but will roll out over several weeks, but early phases of that implementation should also reduce the request times that were originally triggering the 504 timeouts. What you can do: If you're currently affected, we recommend following updates here, and subscribing to the public issue: https://github.com/lemurheavy/coveralls-public/issues/1757 If your issue pattern differs from above, or you suspect a different root cause, reach out to support@coveralls.io, and we'll verify for you.

Elevated 504 Timeout Errors

Aug <var data-var='date'>20</var>, <var data-var='time'>15:21</var> PDT Resolved - We have implemented an intermediate solution and believe this issue has been resolved for now. To fully resolve the root cause, we will need to implement a more long-term solution, which is currently in design. (Please read postmortem for more information.) In the meantime, we will be doing our best to monitor for spikes in traffic from outlier repos and manually respond where our intermediate solution may not mitigate as much as we hope it will. Aug <var data-var='date'>20</var>, <var data-var='time'>12:08</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>20</var>, <var data-var='time'>10:11</var> PDT Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>20</var>, <var data-var='time'>10:11</var> PDT Update - We are continuing to investigate this issue. Aug <var data-var='date'>20</var>, <var data-var='time'>08:48</var> PDT Investigating - We are currently investigating this issue.

More reports of “Website under heavy load”

Aug <var data-var='date'>19</var>, <var data-var='time'>10:26</var> PDT Resolved - We believe we have resolved all intermittent instances of the "This website is under heavy load" 503 error from our HTTP servers. If you should happen to receive that error from here, please let us know at: support@coveralls.io. Aug <var data-var='date'>19</var>, <var data-var='time'>09:24</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>19</var>, <var data-var='time'>08:21</var> PDT Investigating - We are continuing to receive errors of customers receiving "This website is under heavy load" errors from our HTTP servers, even as traffic is normal. We implemented a fix last night that resolved the issue for 6-8 hrs, until we received a new report. We are investigating the issue to identify a permanent fix. In the meantime, if you receive this error, please re-try your request.

Reports of "Website under heavy load" errors

Aug <var data-var='date'>19</var>, <var data-var='time'>01:10</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'>19</var>, <var data-var='time'>00:52</var> PDT Update - We are continuing to monitor for any further issues. Aug <var data-var='date'>18</var>, <var data-var='time'>23:51</var> PDT Update - We are continuing to monitor for any further issues. Aug <var data-var='date'>18</var>, <var data-var='time'>20:56</var> PDT Update - We are continuing to monitor for any further issues. Aug <var data-var='date'>18</var>, <var data-var='time'>15:24</var> PDT Update - We are continuing to monitor for any further issues. Aug <var data-var='date'>18</var>, <var data-var='time'>14:23</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>18</var>, <var data-var='time'>13:25</var> PDT Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>18</var>, <var data-var='time'>12:25</var> PDT Investigating - We are investigating reports of this error which was first seen last week. We believe this is an error being thrown by our HTTP server but has nothing to do with actual website load, which is currently normal. If you receive this error, please re-try your request.

Aug <var data-var='date'>14</var>, <var data-var='time'>07:45</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'>14</var>, <var data-var='time'>07:35</var> PDT Identified - The issue has been identified and a fix is being implemented.

Service unavailable with HTML error page or 500 errors

Aug <var data-var='date'>13</var>, <var data-var='time'>12:24</var> PDT Resolved - Closing this incident as a potential false alarm. Connection stats show no issues and we cannot reproduce any responses of, "This website is under heavy load (queue full)." Nor do we see any in logs from this morning. We now believe the report we received this morning may have actually been encountered yesterday, around the same time as another report, before the incident was resolved. (Confirmed.) Please let us know if you have received the Web or API response, "This website is under heavy load (queue full)," and when you encountered it—which would be especially helpful. Aug <var data-var='date'>13</var>, <var data-var='time'>10:14</var> PDT Identified - We've received reports that some used are receiving Web or API responses of: "This website is under heavy load (queue full)." We have identified the source of the issue and are working to resolve it now.

Is Coveralls down?

Coveralls status is UP

Lastest published events

Increased latency for Large Repos (5k+ source files)

Elevated errors rates

"Website under heavy load" warning

Coveralls Services, Components and Regions

Recent Coveralls events

Increased latency for Large Repos (5k+ source files)

Elevated errors rates

"Website under heavy load" warning

Increased latency for large repos

Increased latency for large repos

Increased latency for large repos