Mar <var data-var='date'> 9</var>, <var data-var='time'>23:33</var> UTC Resolved - We have now resolved this issue. Ranking is now operational, though many cookies may have been expired due to a change pushed by Roblox. Mar <var data-var='date'> 9</var>, <var data-var='time'>04:50</var> UTC Identified - Campfire/Hyra ranking is currently not performing as expected due to a change with how Roblox manages cookies. We are working on a solution.

Ranking System

Jan <var data-var='date'>31</var>, <var data-var='time'>16:30</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>31</var>, <var data-var='time'>14:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>31</var>, <var data-var='time'>11:39</var> UTC Scheduled - During the period 1400 UTC and 1600 UTC we will be migrating our GB services. No service disruption is expected, though it could be a possibility. Customers may request a migration from GB to another pod if they are concerned about any potential downtime. Please contact support to do this.

Migrating GB Pods (44, 1922, 55)

Jan <var data-var='date'>20</var>, <var data-var='time'>00:42</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>19</var>, <var data-var='time'>23:55</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jan <var data-var='date'>19</var>, <var data-var='time'>23:20</var> UTC Update - We have identified the issue and are rolling out a fix. Jan <var data-var='date'>19</var>, <var data-var='time'>22:59</var> UTC Identified - The issue has been identified and a fix is being implemented. Jan <var data-var='date'>19</var>, <var data-var='time'>20:56</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jan <var data-var='date'>19</var>, <var data-var='time'>19:57</var> UTC Investigating - We're currently investigating an issue with our bot network

Issues with bots

Jan <var data-var='date'>14</var>, <var data-var='time'>16:51</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>14</var>, <var data-var='time'>13:51</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>14</var>, <var data-var='time'>13:50</var> UTC Scheduled - We will be undergoing scheduled maintenance during this time.

Member Counter Maintenance

Nov <var data-var='date'>16</var>, <var data-var='time'>16:51</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'>16</var>, <var data-var='time'>16:10</var> UTC Identified - This issue is caused by an upstream internet provider issue. We will provide more information as we receive it.

Delayed or no response on website

Oct <var data-var='date'>26</var>, <var data-var='time'>01:00</var> UTC Completed - The scheduled maintenance has been completed. Oct <var data-var='date'>25</var>, <var data-var='time'>00:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>11</var>, <var data-var='time'>00:00</var> UTC Scheduled - During daylight saving, some unexpected bugs may occur. Our team will be carrying out maintenance on this day to resolve any bugs that could potentially occur. The Campfire Team apologises about any inconvenience this could cause, and will try to keep disruption to a minimum.

Daylight saving

Oct <var data-var='date'>11</var>, <var data-var='time'>19:30</var> UTC Resolved - Roblox has recovered and Campfire Ranking requests are now operational. Oct <var data-var='date'>11</var>, <var data-var='time'>17:57</var> UTC Identified - A Roblox issue causing decreased performance on the platform is impacting performance of Campfire Ranking. Oct <var data-var='date'>11</var>, <var data-var='time'>17:32</var> UTC Investigating - We are currently investigating this issue.

Degraded ranking performance

Oct <var data-var='date'>10</var>, <var data-var='time'>19:47</var> UTC Resolved - Roblox has recovered and Campfire Ranking requests are now operational. Oct <var data-var='date'>10</var>, <var data-var='time'>18:29</var> UTC Update - We're experiencing an elevated level of API errors due to a Roblox outage. The majority of requests are still going through, however a small percentage are failing. Oct <var data-var='date'>10</var>, <var data-var='time'>17:57</var> UTC Identified - Ranking performance is currently degraded. We are looking into this issue and will provide an update shortly.

Sep <var data-var='date'> 5</var>, <var data-var='time'>14:10</var> UTC Completed - The scheduled maintenance has been completed. Sep <var data-var='date'> 5</var>, <var data-var='time'>14:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'> 5</var>, <var data-var='time'>01:47</var> UTC Scheduled - We anticipate the possibility of downtime of less than 10 minutes while we provision a large backbone platform on our campfire.host infrastructure. This may cause some problems with our internal publish subscriptions API between our bot pods, but the downtime will be kept below 10 minutes. We will work diligently and swiftly to ensure that services are kept to a minimal disruption. We apologise about any inconvenience this may cause.

Provisioning Strivo

Aug <var data-var='date'>14</var>, <var data-var='time'>14:24</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>14</var>, <var data-var='time'>14:15</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>14</var>, <var data-var='time'>14:10</var> UTC Update - A fix has been implemented and these changes are now deploying across the CFHQ network. Aug <var data-var='date'>14</var>, <var data-var='time'>14:03</var> UTC Identified - We are working hard to roll out a fix as soon as possible for this security bug.

Security vulnerability

Aug <var data-var='date'>10</var>, <var data-var='time'>00:57</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>10</var>, <var data-var='time'>00:47</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>10</var>, <var data-var='time'>00:45</var> UTC Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>10</var>, <var data-var='time'>00:40</var> UTC Investigating - An issue has automatically been identified with member counters. A team is now investigating the issue.

Member Counters

Aug <var data-var='date'> 7</var>, <var data-var='time'>21:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'> 7</var>, <var data-var='time'>20:24</var> UTC Verifying - Verification is currently underway for the maintenance items. Aug <var data-var='date'> 7</var>, <var data-var='time'>20:01</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'> 5</var>, <var data-var='time'>00:53</var> UTC Scheduled - We will be carrying out scheduled maintenance on our group shout proxies to remove “Powered by campfirehq.net” from the footer, making this product entirely white labelled. We anticipate this maintenance to last no longer than 60 minutes. Updates will be posted as required. We understand that this may cause some inconvenience for customers, and will work as quickly and as diligently as possible. Thank you for your ongoing support.

Update to shout proxies

Jul <var data-var='date'>17</var>, <var data-var='time'>21:43</var> UTC Resolved - We're marking this as resolved, further information can be found at https://www.cloudflarestatus.com/ Jul <var data-var='date'>17</var>, <var data-var='time'>21:32</var> UTC Investigating - There is currently an outage across Campfire products and services due to an upstream internet issue.

All systems unavailable

Jul <var data-var='date'>17</var>, <var data-var='time'>12:44</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>17</var>, <var data-var='time'>11:54</var> UTC Investigating - We are currently investigating this issue.

Ranking services down

Jul <var data-var='date'>16</var>, <var data-var='time'>22:19</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>16</var>, <var data-var='time'>10:31</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'>16</var>, <var data-var='time'>10:29</var> UTC Investigating - We are currently investigating this issue. Jul <var data-var='date'>16</var>, <var data-var='time'>10:20</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>16</var>, <var data-var='time'>08:53</var> UTC Investigating - We are currently investigating an issue regarding member counters not sending correct or at a reduced interval. As this product continues to grow in size, we are finding it increasingly difficult to manage the 1,728,000+ requests per day we send to Roblox. We are working closely with our DevOps Team to resolve the situation as soon as possible and we will update you as required. We apologise about the inconvenience.

Member Counters not operating as expected.

Jul <var data-var='date'>12</var>, <var data-var='time'>13:25</var> UTC Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>12</var>, <var data-var='time'>12:25</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>12</var>, <var data-var='time'>12:22</var> UTC Scheduled - We are upgrading the resources on Pod 212. Services on Pod 212 will be down whilst this upgrade occurs.

Pod 212 - Upgrade

Jul <var data-var='date'> 1</var>, <var data-var='time'>20:49</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 1</var>, <var data-var='time'>18:56</var> UTC Monitoring - The site is now resumed. We are monitoring and will update if the situation changes. Jul <var data-var='date'> 1</var>, <var data-var='time'>18:48</var> UTC Identified - Issues are still occuring. We will update you as required. Jul <var data-var='date'> 1</var>, <var data-var='time'>18:44</var> UTC Monitoring - The site now back up. Our team are continuing to monitor and will keep you updated should issueso occur again. Jul <var data-var='date'> 1</var>, <var data-var='time'>18:42</var> UTC Update - The site is now coming back online slowly. Jul <var data-var='date'> 1</var>, <var data-var='time'>18:37</var> UTC Identified - Roblox.com is currently down.

Roblox.com down

Jun <var data-var='date'>30</var>, <var data-var='time'>17:27</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>30</var>, <var data-var='time'>17:26</var> UTC Verifying - Maintenance tasks completed, verifying everything working as expected. Jun <var data-var='date'>30</var>, <var data-var='time'>17:24</var> UTC Update - Pod 1024 is now being moved. Jun <var data-var='date'>30</var>, <var data-var='time'>17:11</var> UTC Update - Pod 525 is now confirmed online. Jun <var data-var='date'>30</var>, <var data-var='time'>17:10</var> UTC Update - Pod 525 is now moving. Jun <var data-var='date'>30</var>, <var data-var='time'>17:06</var> UTC Update - Pod 666 and Pod 3662 are now fully operational. Jun <var data-var='date'>30</var>, <var data-var='time'>17:04</var> UTC Update - Pod 666 and 3662 are now being moved. Jun <var data-var='date'>30</var>, <var data-var='time'>17:02</var> UTC Update - Pod 212 is now restored and fully operation. We are now doing Pod 666. Jun <var data-var='date'>30</var>, <var data-var='time'>16:51</var> UTC Update - Pod 212 is currently being transferred. All bots on pod 212 are currently offline. Jun <var data-var='date'>30</var>, <var data-var='time'>16:46</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>30</var>, <var data-var='time'>16:01</var> UTC Update - We'll be moving all pods over to the new provider in 45 minutes. Some downtime is expected. Jun <var data-var='date'>30</var>, <var data-var='time'>16:01</var> UTC Update - We will be undergoing scheduled maintenance during this time. Jun <var data-var='date'>28</var>, <var data-var='time'>02:14</var> UTC Scheduled - We will be moving our Pod 212 to a new location during this time. We will try to keep service downtime to a minimum, during this time all customers using custom bots on Pod 212 will be re-routed to temporary Pod 790, located in Amsterdam. We apologise sincerely about any inconvenience this may cause to customers. Please also note that the following legacy bots will also be affected: - Pastriez

Centralised Provider Change

Jun <var data-var='date'>22</var>, <var data-var='time'>17:49</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>22</var>, <var data-var='time'>16:58</var> UTC Update - We are working hard on a fix. Jun <var data-var='date'>22</var>, <var data-var='time'>10:08</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>22</var>, <var data-var='time'>09:44</var> UTC Update - We’re going to test a fix in production now. The only caveat of this is that there will be an increased delay between the counting - i.e. a longer amount of time will pass before we check for new members. Jun <var data-var='date'>22</var>, <var data-var='time'>00:49</var> UTC Identified - There is currently an issue where member counters may jump back and forth between two numbers. This is caused by a problem in the Roblox group count system, and we are looking into fixes for this problem. This service continues to operate as normal.

Jun <var data-var='date'>20</var>, <var data-var='time'>21:51</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>20</var>, <var data-var='time'>18:44</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>20</var>, <var data-var='time'>18:42</var> UTC Update - A fix will be rolling out very shortly. Jun <var data-var='date'>20</var>, <var data-var='time'>18:35</var> UTC Identified - Issue has been identified and a fix is being implemented. Jun <var data-var='date'>20</var>, <var data-var='time'>16:22</var> UTC Update - Our DevOps Team are continuing to investigate the issue. Jun <var data-var='date'>20</var>, <var data-var='time'>12:45</var> UTC Update - DevOps Team are now working on a solution to this problem. Jun <var data-var='date'>20</var>, <var data-var='time'>02:59</var> UTC Update - All hooks and the ability to create and manage hooks has been disabled until DevOps Team becomes available. Apologies about the inconvenience. Jun <var data-var='date'>20</var>, <var data-var='time'>02:55</var> UTC Update - All DevOps team are currently unavailable. No expectation of resolution until DevOps Team become available again in 5-10 hours. Jun <var data-var='date'>20</var>, <var data-var='time'>02:53</var> UTC Investigating - We are currently investigating this issue.

Member counters not operating

Jun <var data-var='date'>19</var>, <var data-var='time'>09:44</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>18</var>, <var data-var='time'>23:45</var> UTC Update - This issue may still be present for some users. We advise clearing your cache or using Ctrl + Shift + R / ⌘ + Shift + R Jun <var data-var='date'>18</var>, <var data-var='time'>23:41</var> UTC Update - We expect this to be resolved for all users by 00:00 UTC. Jun <var data-var='date'>18</var>, <var data-var='time'>23:40</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.

Issues with caching CDN

Jun <var data-var='date'> 7</var>, <var data-var='time'>00:41</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'> 7</var>, <var data-var='time'>00:39</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 7</var>, <var data-var='time'>00:37</var> UTC Identified - The issue was identified as a type error. We’re now rolling out a patch. Jun <var data-var='date'> 7</var>, <var data-var='time'>00:34</var> UTC Investigating - We are currently investigating this issue.

Billing System Not Activating Bots

May <var data-var='date'>16</var>, <var data-var='time'>12:00</var> UTC Completed - The scheduled maintenance has been completed. May <var data-var='date'>16</var>, <var data-var='time'>10:01</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'> 1</var>, <var data-var='time'>19:35</var> UTC Scheduled - Zendesk will be performing DNS maintenance during a scheduled window listed below. No downtime is expected. Details: https://support.zendesk.com/hc/en-us/articles/360047691573-Scheduled-Maintenance-May-16-2020-DNS-Change-All-Pods

Zendesk - DNS Change

May <var data-var='date'> 6</var>, <var data-var='time'>15:43</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 6</var>, <var data-var='time'>13:08</var> UTC Monitoring - A fix has been deployed. We are now monitoring the fix. May <var data-var='date'> 6</var>, <var data-var='time'>11:40</var> UTC Update - Our team are working on implementing rate limiting technology. We won’t ever hard rate limit, we’ll simply slow down your requests in transit. This is to prevent us from sending too many requests to Roblox’s servers. We will publish documentation to our Developer Docs when this has been implemented. Service will resume once this has been implemented. Thank you for your patience and cooperation. May <var data-var='date'> 5</var>, <var data-var='time'>20:36</var> UTC Identified - We've identified the root cause as a denial of service attack on our ranking service. We are taking security precautions to resolve the issue. May <var data-var='date'> 5</var>, <var data-var='time'>20:01</var> UTC Update - We have no ETA of when this can be resolved. The issues are beyond the scope of Campfire, and are with our providers. May <var data-var='date'> 5</var>, <var data-var='time'>19:56</var> UTC Update - We are continuing to investigate this issue. May <var data-var='date'> 5</var>, <var data-var='time'>19:56</var> UTC Update - We've temporarily paused this service as we are facing issues on our backend database. We work with our team to resolve the issue. May <var data-var='date'> 5</var>, <var data-var='time'>19:54</var> UTC Update - We are continuing to investigate this issue. May <var data-var='date'> 5</var>, <var data-var='time'>19:40</var> UTC Investigating - We're currently investigating an issue with Campfire ranking and 429 errors. We're working closely with our DevOps Team to resolve the issues as quickly as possible.

Ranking issues

May <var data-var='date'> 4</var>, <var data-var='time'>12:15</var> UTC Completed - The scheduled maintenance has been completed. Thank you for your patience. May <var data-var='date'> 4</var>, <var data-var='time'>12:00</var> UTC Update - We're now temporarily pausing our hooks. May <var data-var='date'> 4</var>, <var data-var='time'>09:45</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'> 4</var>, <var data-var='time'>08:57</var> UTC Scheduled - We are upgrading the security of the group shout proxy and member counters to support two groups with the same ID. Previously, it would cause an error when doing it and would actually edit another users configuration.

Is Campfire HQ down?

Campfire HQ status is UP

Lastest published events

Ranking System

Migrating GB Pods (44, 1922, 55)

Issues with bots

Campfire HQ Services, Components and Regions

Recent Campfire HQ events

Ranking System

Migrating GB Pods (44, 1922, 55)

Issues with bots

Member Counter Maintenance

Delayed or no response on website

Daylight saving