Aug <var data-var='date'> 9</var>, <var data-var='time'>08:46</var> PDT Resolved - Errors have returned to normal levels. Fix confirmed. Resolving. Aug <var data-var='date'> 9</var>, <var data-var='time'>07:53</var> PDT Monitoring - Throttling adjustments are in place. Errors subsiding. Monitoring. Aug <var data-var='date'> 9</var>, <var data-var='time'>06:50</var> PDT Identified - Bots/crawlers are overwhelming the service. Adjusting throttling. Aug <var data-var='date'> 9</var>, <var data-var='time'>06:38</var> PDT Investigating - We are currently investigating this issue.

Elevated error rates for badge service

May <var data-var='date'>20</var>, <var data-var='time'>08:55</var> PDT Resolved - Configuration & scaling improvements seem to be absorbing the traffic spikes more comfortably now. Resolving. May <var data-var='date'>20</var>, <var data-var='time'>06:35</var> PDT Monitoring - The number of open connections to each server was reaching our configured limit. We have raised the limit. Monitoring. May <var data-var='date'>20</var>, <var data-var='time'>06:20</var> PDT Investigating - We're seeing unusual traffic to our repositories that are causing elevated errors and timeouts. We've investigating possible ways to mitigate this.

Investigating elevated repo errors

Mar <var data-var='date'>25</var>, <var data-var='time'>01:31</var> PDT Resolved - Jobs processing normally, and outstanding work has been completed. Resolving. Mar <var data-var='date'>25</var>, <var data-var='time'>01:03</var> PDT Monitoring - Looks like some of the misconfiguration issues persisted from yesterday. A few of the queues were not processed, which prevented account updates from propagating to repositories. We have made a fix, and are now processing the backlog. Mar <var data-var='date'>25</var>, <var data-var='time'>00:44</var> PDT Investigating - We are again seeing errors & delays for uploads. Investigating.

Elevated errors for uploads and asynchronous work

Mar <var data-var='date'>24</var>, <var data-var='time'>11:40</var> PDT Resolved - Production-specific misconfiguration caused our async workers to enter a crash-restart loop. We've fixed the issue, and we will be looking into how to catch these issues earlier in the release process. Resolving. Mar <var data-var='date'>24</var>, <var data-var='time'>09:34</var> PDT Investigating - We are investigating delays in processing of uploads and other asynchronous work.

Job processing delays & errors

Mar <var data-var='date'>16</var>, <var data-var='time'>10:23</var> PDT Resolved - Resolving the incident to continue the investigation and fix offline. Mar <var data-var='date'>16</var>, <var data-var='time'>09:53</var> PDT Monitoring - We had a temporary 10x traffic increase on one of our backend services, which lead to elevated timeouts and errors. Error rates have now returned to normal while we continue to look for the root cause of that traffic spike. Mar <var data-var='date'>16</var>, <var data-var='time'>08:41</var> PDT Investigating - We are currently investigating this issue.

Elevated errors

Feb <var data-var='date'>21</var>, <var data-var='time'>22:51</var> PST Resolved - Operations returned to normal. We will address the performance issue through our development cycle. Feb <var data-var='date'>21</var>, <var data-var='time'>21:12</var> PST Monitoring - New code with an inefficient query in a critical path deployed. We have rolled it back - errors returning to normal. Feb <var data-var='date'>21</var>, <var data-var='time'>20:59</var> PST Investigating - Investigating elevated error rates

Feb <var data-var='date'> 9</var>, <var data-var='time'>08:21</var> PST Resolved - We've tracked the issue down to a frequent DB update and applied a fix to reduce it. Resolving the incident while we will keep monitoring the fix. Feb <var data-var='date'> 9</var>, <var data-var='time'>03:30</var> PST Monitoring - Unusual traffic spike seems to have temporarily overwhelmed one of our API endpoints. That spike has subsided, error/timeout rates are back to normal. We are monitoring while we continue to investigate. Feb <var data-var='date'> 9</var>, <var data-var='time'>03:21</var> PST Investigating - We are investigating elevated error and timeout rates.

Elevated errors & timeouts

Oct <var data-var='date'>20</var>, <var data-var='time'>21:39</var> PDT Resolved - Upstream issue resolved. If you’re still having availability issues, don’t hesitate to contact us directly. Oct <var data-var='date'>20</var>, <var data-var='time'>11:04</var> PDT Monitoring - We are continuing to monitor the upstream issue. Oct <var data-var='date'>20</var>, <var data-var='time'>08:59</var> PDT Identified - We are reopening the incident due new availability issues related to AWS outage.

AWS outage effects

Oct <var data-var='date'>20</var>, <var data-var='time'>03:41</var> PDT Resolved - Upstream issue has been resolved. We are back to normal. Resolving. Oct <var data-var='date'>20</var>, <var data-var='time'>03:15</var> PDT Monitoring - As many others, we have been affected by the AWS outage. Particularly, our Git/Build service could not accept pushes and trigger new builds. Currently, upstream errors and our errors have subsided, we are monitoring toward a resolution. Note: We were unable to update this status page because this service was also affected.

Sep <var data-var='date'>18</var>, <var data-var='time'>10:28</var> PDT Resolved - It appears a misplaced space in a configuration update of caused the configured library to throw exceptions for some of the requests. We've fixed the configuration and will be patching the library to ensure this is handled more gracefully. Resolving. Sep <var data-var='date'>18</var>, <var data-var='time'>09:13</var> PDT Monitoring - We've found that the issue was triggered by a recent configuration update. We've rolled back the changes, and the error rates have returned to normal levels. Checking further into the changes that caused the failure. Sep <var data-var='date'>18</var>, <var data-var='time'>08:37</var> PDT Investigating - We are seeing elevated error rates. Investigating.

Elevated error rates

Jun <var data-var='date'>12</var>, <var data-var='time'>14:01</var> PDT Resolved - Upstream incident has been resolved, and no problems otherwise. Resolving. Jun <var data-var='date'>12</var>, <var data-var='time'>12:20</var> PDT Identified - We are seeing increased timeouts for proxied requests to the public npm registry. We are tracking the upstream issue with npmjs.org.

Timeouts for proxied npm registry

Jun <var data-var='date'>10</var>, <var data-var='time'>20:14</var> PDT Resolved - We are resolving this as we are no longer seeing customer-facing errors. We'll continue to monitor both our and the platform's status to address any issues that may arise. Jun <var data-var='date'>10</var>, <var data-var='time'>12:20</var> PDT Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>10</var>, <var data-var='time'>12:20</var> PDT Update - Although some platform instability persists, we have processed most of the asynchronous task backlog (uploads, emails, etc.). Most functionality has been restored. We will continue monitoring the status of the platform until resolution. If you're still experiencing delays or issues with your account, please contact support@gemfury.com with full details. Jun <var data-var='date'>10</var>, <var data-var='time'>10:27</var> PDT Monitoring - We were able to start some of the workers. Starting to work through the async task backlog. Jun <var data-var='date'>10</var>, <var data-var='time'>08:53</var> PDT Update - Our asynchronous workers are not running due to platform issues, and thus cannot process uploads. We are waiting for updates from our platform provider. Jun <var data-var='date'>10</var>, <var data-var='time'>05:51</var> PDT Investigating - We are currently investigating this issue.

Platform issues causing upload failures

Nov <var data-var='date'>20</var>, <var data-var='time'>00:24</var> PST Resolved - The fix was effective. Service fully restored. Nov <var data-var='date'>19</var>, <var data-var='time'>22:48</var> PST Monitoring - We've deployed an update that caused elevated errors due to a caching error that didn't surface during testing. We've rolled back the update, implemented a fix, and rolled out the fixed update. Monitoring.

Elevated errors after deployment

Aug <var data-var='date'>17</var>, <var data-var='time'>02:42</var> PDT Resolved - Request and error rates returning to normal levels. We will continue to monitor. Resolving. Aug <var data-var='date'>17</var>, <var data-var='time'>02:14</var> PDT Monitoring - A routine infrastructure update misconfigured our load balancers. We've updated the load balancers and updated DNS settings. It may take time for changes to propagate to clients. Monitoring. Aug <var data-var='date'>17</var>, <var data-var='time'>01:59</var> PDT Investigating - Investigating custom domains and uploads outage.

Custom domains and uploads outage

Jul <var data-var='date'> 6</var>, <var data-var='time'>07:57</var> PDT Resolved - Back to normal. Resolving. Jul <var data-var='date'> 6</var>, <var data-var='time'>07:28</var> PDT Monitoring - Shortage of resources prevented build jobs from being scheduled. Scaling up nodes seems to have fixed the issue. Monitoring. Jul <var data-var='date'> 6</var>, <var data-var='time'>07:12</var> PDT Investigating - Investigating build failures

Build failures

Apr <var data-var='date'>20</var>, <var data-var='time'>10:01</var> PDT Resolved - Clock drift resulted in authentication failure between Git server and builder nodes. Correcting the time resolved the sporadic failures. We will investigate why nodes don't automatically synchronize their clocks. Resolving the immediate issue. Apr <var data-var='date'>20</var>, <var data-var='time'>08:32</var> PDT Investigating - We are investigating an issue with failing builds via `git push`

Failing builds via Git

Sep <var data-var='date'>23</var>, <var data-var='time'>19:48</var> PDT Completed - Successfully completed maintenance. Resolving. Sep <var data-var='date'>23</var>, <var data-var='time'>19:31</var> PDT Verifying - Maintenance completed. Monitoring for anomalies. Sep <var data-var='date'>23</var>, <var data-var='time'>19:01</var> PDT Update - Starting maintenance Sep <var data-var='date'>23</var>, <var data-var='time'>19:01</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>16</var>, <var data-var='time'>08:24</var> PDT Scheduled - We will be performing database maintenance that will cause downtime during this period.

Brief downtime for DB maintenance

Sep <var data-var='date'>15</var>, <var data-var='time'>12:48</var> PDT Resolved - Cluster access restored. Git repo building issues have been resolved. Sep <var data-var='date'>15</var>, <var data-var='time'>12:45</var> PDT Identified - Payment issue with our cluster provider caused temporary API suspension. We've fixed the payment, and waiting for restoration of cluster access. Sep <var data-var='date'>15</var>, <var data-var='time'>12:34</var> PDT Investigating - Investigating "git push" build issues

Git-push build errors

Aug <var data-var='date'>18</var>, <var data-var='time'>17:51</var> PDT Resolved - Everything returned to operating normally. Resolving. Aug <var data-var='date'>18</var>, <var data-var='time'>17:40</var> PDT Monitoring - A cache instance has failed triggering automatic failover. Error rates returning to normal. Aug <var data-var='date'>18</var>, <var data-var='time'>17:34</var> PDT Investigating - We are investigating elevated error rates

Aug <var data-var='date'> 8</var>, <var data-var='time'>03:07</var> PDT Resolved - We've found a caching bug that was introduced by the latest deployment. It will not be present in future builds. Resolving. Aug <var data-var='date'> 8</var>, <var data-var='time'>00:59</var> PDT Monitoring - We've rolled back most recent deployment. Still debugging. Aug <var data-var='date'> 7</var>, <var data-var='time'>21:57</var> PDT Investigating - Investigating latency & error spike

Elevated error responses

Jun <var data-var='date'>29</var>, <var data-var='time'>09:00</var> PDT Resolved - Earlier today, we deployed a bug that exposed a legacy system to excessive traffic, overwhelming our core Redis instance and causing cascading failures in certain functionality. In the past, we occasionally used a legacy internal system to capture debug information for rare data states, aiding in tracking and reproducing customer issues. This information was stored in our core non-volatile Redis instance. While this approach had worked for rare conditions and low-traffic code paths, this incident occurred due to mistakenly adding such tracking to heavily-trafficked functionality. May 29th, 12:43 UTC - Deployment of a release with the tracking bug The release containing the tracking bug passed preflight and was deployed to production. Initially, everything appeared stable. However, the utilization of our non-volatile Redis, which usually hovers below 5%, slowly started to increase. Unfortunately, this increase went unnoticed. May 29th, 15:55 UTC - Redis storage reached maximum utilization When the non-volatile Redis storage reached 100% utilization, write operations began receiving "OOM command not allowed" error responses, resulting in 500 errors for certain user-facing APIs. Most read operations were successful, but not all. Regrettably, the tracking code was present in the API layer servicing the Dashboard and the CLI, causing errors for those read operations. Worse, the error rate remained low enough to not trigger any alarms. May 29th, 23:04 UTC - Redis instance cleaned to restore service Customer service noticed an increase in error reports and promptly notified engineering to investigate. Engineering quickly identified the issue and cleared excess data from Redis, restoring service. May 29th, 23:40 UTC - Fix deployed to remove the tracking bug We deployed a fix to remove the tracking bug. Further steps Later in the day, we removed the legacy tracking system and migrated that functionality to use standard error and metrics tracking. This step will prevent similar space issues in the future and consolidate our monitoring infrastructure. Moving forward, we will implement more alarms for storage utilization and introduce more fine-grained tracking of error rates.

Partial service failure due to non-volatile Redis saturation

May <var data-var='date'>20</var>, <var data-var='time'>22:31</var> PDT Completed - The scheduled maintenance has been completed. May <var data-var='date'>20</var>, <var data-var='time'>22:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>20</var>, <var data-var='time'>21:35</var> PDT Scheduled - We will be performing some maintenance that may cause minimal downtime.

May <var data-var='date'>13</var>, <var data-var='time'>16:00</var> PDT Completed - The scheduled maintenance has been completed. May <var data-var='date'>13</var>, <var data-var='time'>15:02</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>12</var>, <var data-var='time'>19:54</var> PDT Scheduled - Our database will be undergoing routine maintenance. We may experience elevated errors on all services.

Feb <var data-var='date'> 8</var>, <var data-var='time'>10:33</var> PST Resolved - Everything confirmed operational. We will resolve this incident, but will continue investigating the cause of the outage. Feb <var data-var='date'> 8</var>, <var data-var='time'>10:15</var> PST Monitoring - We've paused traffic and restarted services. Errors subsiding, requests being serviced normally. Looking into what happened. Feb <var data-var='date'> 8</var>, <var data-var='time'>09:59</var> PST Investigating - We're seeing elevated errors on all components. Investigating.

Dec <var data-var='date'>20</var>, <var data-var='time'>19:17</var> PST Resolved - This incident has been resolved. Dec <var data-var='date'>20</var>, <var data-var='time'>17:15</var> PST Monitoring - Fix applied. Error rates recovering. Monitoring. Dec <var data-var='date'>20</var>, <var data-var='time'>16:59</var> PST Identified - Incorrect cache configuration is causing high latencies and timeouts. Deploying a fix. Dec <var data-var='date'>20</var>, <var data-var='time'>16:52</var> PST Investigating - We are currently investigating this issue.

Is Gemfury down?

Gemfury status is UP

Lastest published events

Elevated error rates for badge service

Investigating elevated repo errors

Elevated errors for uploads and asynchronous work

Gemfury Services, Components and Regions

Recent Gemfury events

Elevated error rates for badge service

Investigating elevated repo errors

Elevated errors for uploads and asynchronous work

Job processing delays & errors

Elevated errors

Elevated errors