Type: Maintenance
 Duration: 30 minutes
 Affected Components: 
 Oct <var data-var='date'> 30</var>, <var data-var='time'>12:20:00</var> GMT+0 Completed -
 Maintenance has completed successfully.
Oct <var data-var='date'> 30</var>, <var data-var='time'>11:58:31</var> GMT+0 Identified -
 Maintenance has completed successfully..
Oct <var data-var='date'> 30</var>, <var data-var='time'>11:50:01</var> GMT+0 Identified -
 Maintenance is now in progress.
Oct <var data-var='date'> 30</var>, <var data-var='time'>11:45:51</var> GMT+0 Identified -
 We are planning for a scheduled maintenance during that time..

TingleFlow down for upgrades

Type: Incident
 Duration: 6 days, 22 hours and 21 minutes
 Affected Components: 
 Oct <var data-var='date'> 29</var>, <var data-var='time'>11:01:22</var> GMT+0 Resolved -
 This incident has been resolved..
Oct <var data-var='date'> 22</var>, <var data-var='time'>12:40:04</var> GMT+0 Identified -
 We&#039;ve identified the cause of this issue and are working on a fix..

TingleFlow Radio and Sparks unavailable for some users

Type: Maintenance
 Duration: 3 hours and 50 minutes
 Affected Components: 
 Oct <var data-var='date'> 22</var>, <var data-var='time'>12:50:00</var> GMT+0 Completed -
 Maintenance has completed successfully.
Oct <var data-var='date'> 22</var>, <var data-var='time'>11:57:10</var> GMT+0 Identified -
 We&#039;re continuing to work on this migration..
Oct <var data-var='date'> 22</var>, <var data-var='time'>10:33:15</var> GMT+0 Identified -
 We&#039;re continuing to work on this migration. We would like to thank everyone for their patience..
Oct <var data-var='date'> 22</var>, <var data-var='time'>09:20:45</var> GMT+0 Identified -
 Migration beginning.
Oct <var data-var='date'> 22</var>, <var data-var='time'>09:00:01</var> GMT+0 Identified -
 Maintenance is now in progress.
Oct <var data-var='date'> 21</var>, <var data-var='time'>12:52:22</var> GMT+0 Identified -
 We&#039;re migrating TingleFlow&#039;s direct messaging media to a new content delivery service (CDN) as part of our ongoing infrastructure improvements. This migration is intended to improve how media shared through direct and group messages is delivered and accessed.

During the migration, some messaging media may be temporarily unavailable or experience interruptions while content is transferred and the new delivery configuration is put in place. We&#039;ll provide updates as the migration progresses and confirm once the process is complete..

Direct Messaging CDN Migration

Type: Maintenance
 Duration: 3 hours and 44 minutes
 Affected Components: 
 Oct <var data-var='date'> 21</var>, <var data-var='time'>12:44:18</var> GMT+0 Completed -
 Maintenance has completed successfully..
Oct <var data-var='date'> 21</var>, <var data-var='time'>12:27:14</var> GMT+0 Identified -
 **We expect maintenance to be completed within the next 10-20 minutes.**.
Oct <var data-var='date'> 21</var>, <var data-var='time'>11:47:37</var> GMT+0 Identified -
 We ran into an issue and need to extend the maintenance by 30 minutes after the expected finish time. We&#039;re incredibly sorry about this inconvenience..
Oct <var data-var='date'> 21</var>, <var data-var='time'>11:16:54</var> GMT+0 Identified -
 Migration is still underway..
Oct <var data-var='date'> 21</var>, <var data-var='time'>10:40:25</var> GMT+0 Identified -
 We continue to work on the migration of our CDN.
Oct <var data-var='date'> 21</var>, <var data-var='time'>10:20:59</var> GMT+0 Identified -
 The migration is now underway. We&#039;re beginning to transfer TingleFlow&#039;s content to our new content delivery service. We&#039;ll provide further updates as the upgrade progresses..
Oct <var data-var='date'> 21</var>, <var data-var='time'>09:00:01</var> GMT+0 Identified -
 **TingleFlow is currently undergoing scheduled infrastructure maintenance and is temporarily unavailable to all users.**

The maintenance period has officially begun as we proceed with migrating TingleFlow&#039;s content to a new content delivery service (CDN). This migration is part of our ongoing infrastructure improvements and involves transitioning our content delivery systems to new infrastructure.

During this time, access to TingleFlow and its features will be unavailable while the migration is carried out. This includes accessing the platform, watching content, interacting with creators, and using other TingleFlow services that depend on the affected infrastructure.

Our team is working to complete the migration as smoothly as possible. Once the transition is complete, we will verify that the relevant systems are operating as expected before restoring access to the platform.

We understand that this interruption may be inconvenient, particularly for users who regularly visit TingleFlow to watch content or interact with the community. We apologise for the disruption and appreciate your patience while we carry out this work.

**What happens next?**

We will continue to publish updates on this page as maintenance progresses. Once the migration has been completed and TingleFlow is available again, we will post a final update confirming that service has been restored.

At this stage, please refer to this status page for the latest information regarding the maintenance. We recommend checking back periodically for updates rather than repeatedly attempting to access the platform while the maintenance is ongoing.

Thank you for your patience and understanding as we continue improving the infrastructure that powers TingleFlow..
Oct <var data-var='date'> 21</var>, <var data-var='time'>08:50:03</var> GMT+0 Identified -
 Scheduled maintenance is due to begin in approximately 10 minutes. During this time, TingleFlow will be temporarily unavailable to all users while we carry out essential infrastructure improvements.

We apologise for any inconvenience this may cause and appreciate your patience while we work to improve TingleFlow. Please continue to monitor this status page for further updates..
Oct <var data-var='date'> 21</var>, <var data-var='time'>01:00:11</var> GMT+0 Identified -
 TingleFlow will be temporarily unavailable to all users while we migrate our content to a new content delivery service (CDN). This migration is part of our ongoing infrastructure improvements and is intended to improve how content is delivered across the platform.

During the migration, TingleFlow and most of its features will be inaccessible until the process is complete. We will post updates on this page as the migration progresses and confirm once all services have been restored.

_We encourage all users to learn more about this here:_ &lt;https://help.tingleflow.com/hc/en-us/platform/infrastructure-improvements-at-tingleflow&gt;

Thank you for your patience while we carry out this maintenance..

Content Delivery Migration

Type: Incident
 Duration: 13 hours and 18 minutes
 Affected Components: 
 Oct <var data-var='date'> 20</var>, <var data-var='time'>22:44:24</var> GMT+0 Resolved -
 This incident has been resolved..
Oct <var data-var='date'> 20</var>, <var data-var='time'>11:49:07</var> GMT+0 Monitoring -
 A fix has been implemented and we are seeing excellent recovery. Some services may load slower than usual..
Oct <var data-var='date'> 20</var>, <var data-var='time'>09:26:25</var> GMT+0 Investigating -
 Users connecting to TingleFlow through Google Cloud’s London region (europe-west2) may be unable to connect. Google has acknowledged the issue, and we’ll provide updates as more information becomes available. They have not yet provided any time estimate for recovery. We&#039;re keeping an eye on this and we apologise for the inconvenience this is causing. This is an issue with a third-party platform which TingleFlow has no control of..

TingleFlow Accessibility Issues in Europe

Type: Incident
 Duration: 1 hour and 40 minutes
 Affected Components: 
 Oct <var data-var='date'> 20</var>, <var data-var='time'>00:59:00</var> GMT+0 Resolved -
 This incident has been resolved..
Oct <var data-var='date'> 19</var>, <var data-var='time'>23:36:00</var> GMT+0 Investigating -
 We are currently investigating this incident..

Email delivery issues

Type: Incident
 Duration: 27 minutes
 Affected Components: 
 Oct <var data-var='date'> 19</var>, <var data-var='time'>00:28:53</var> GMT+0 Resolved -
 This incident has been resolved..
Oct <var data-var='date'> 19</var>, <var data-var='time'>00:24:40</var> GMT+0 Monitoring -
 We have deployed a fix and are seeing significant recovery..
Oct <var data-var='date'> 19</var>, <var data-var='time'>00:02:22</var> GMT+0 Investigating -
 We are currently investigating this incident. Pages may show errors and some videos may display as having no views or likes. We&#039;re working on a fix for this issue..

Website issues

Type: Incident
 Duration: 11 minutes
 Affected Components: 
 Oct <var data-var='date'> 16</var>, <var data-var='time'>22:28:05</var> GMT+0 Resolved -
 TingleFlow has returned to a healthy state. Incident resolved..
Oct <var data-var='date'> 16</var>, <var data-var='time'>22:25:16</var> GMT+0 Monitoring -
 We implemented a fix and are currently monitoring the result..
Oct <var data-var='date'> 16</var>, <var data-var='time'>22:16:44</var> GMT+0 Investigating -
 We are currently investigating this incident..

Degraded performance on some TingleFlow services

Type: Incident
 Duration: 10 minutes
 Affected Components: 
 Oct <var data-var='date'> 12</var>, <var data-var='time'>10:32:45</var> GMT+0 Postmortem -
 This was a deployment issue. We reversed it and it mitigated all errors..
Oct <var data-var='date'> 12</var>, <var data-var='time'>10:07:23</var> GMT+0 Resolved -
 This incident has been resolved..
Oct <var data-var='date'> 12</var>, <var data-var='time'>10:04:38</var> GMT+0 Monitoring -
 We implemented a fix and are currently monitoring the result. The site is returning to normal..
Oct <var data-var='date'> 12</var>, <var data-var='time'>10:03:03</var> GMT+0 Identified -
 We have identified the root cause of the issue and are deploying a fix.
Oct <var data-var='date'> 12</var>, <var data-var='time'>09:57:21</var> GMT+0 Investigating -
 We are currently investigating this incident..

Site deployment issues

Type: Incident
 Duration: 1 hour and 54 minutes
 Affected Components: 
 Oct <var data-var='date'> 10</var>, <var data-var='time'>00:30:00</var> GMT+0 Resolved -
 This incident has been resolved..
Oct <var data-var='date'> 9</var>, <var data-var='time'>23:12:45</var> GMT+0 Identified -
 We have identified the issue and are working on a fix..
Oct <var data-var='date'> 9</var>, <var data-var='time'>22:36:21</var> GMT+0 Investigating -
 We are currently investigating this incident. Some aspects of the website are experiencing issues and may not load..

Issues with TingleFlow

Type: Incident
 
 Affected Components: 
 Oct <var data-var='date'> 5</var>, <var data-var='time'>06:38:38</var> GMT+0 Resolved -
 Help Center is back up. This incident was automatically resolved by Instatus monitoring..
Oct <var data-var='date'> 4</var>, <var data-var='time'>16:58:00</var> GMT+0 Investigating -
 We are continuing to investigate this issue..
Oct <var data-var='date'> 4</var>, <var data-var='time'>12:24:08</var> GMT+0 Investigating -
 Help Center is down at the moment. This incident was automatically created by Instatus monitoring..

Help Center API Response Time is back up

Type: Incident
 Duration: 21 minutes
 Affected Components: 
 Oct <var data-var='date'> 4</var>, <var data-var='time'>11:53:19</var> GMT+0 Resolved -
 Scheduled maintenance has been completed..
Oct <var data-var='date'> 4</var>, <var data-var='time'>11:40:14</var> GMT+0 Identified -
 Scheduled maintenance has begun..
Oct <var data-var='date'> 4</var>, <var data-var='time'>11:32:05</var> GMT+0 Identified -
 This is a notice to let you know that planned maintenance is coming up. It’s done regularly to make sure TingleFlow services are even more reliable, fast, and safe. As a result, the TingleFlow website may experience some downtime during this timeframe..

TingleFlow Website Maintenance (Begins at 10:40 PM, AEDT)

Type: Incident
 Duration: 36 minutes
 Affected Components: 
 Sep <var data-var='date'> 16</var>, <var data-var='time'>11:17:40</var> GMT+0 Resolved -
 This incident has been resolved..
Sep <var data-var='date'> 16</var>, <var data-var='time'>10:54:53</var> GMT+0 Monitoring -
 We implemented a fix and the website appears to be operational again. We will continue to monitor..
Sep <var data-var='date'> 16</var>, <var data-var='time'>10:51:34</var> GMT+0 Identified -
 We have identified the issue and are working on a fix..
Sep <var data-var='date'> 16</var>, <var data-var='time'>10:41:33</var> GMT+0 Investigating -
 We are investigating an issue where users may see errors while connecting to Tingleflow, Tingleflow creator hub, Tingleflow ad manager, and help center. Stay tuned..

Issues connecting to Tingleflow

Type: Maintenance
 Duration: 1 hour and 20 minutes
 Affected Components: 
 Sep <var data-var='date'> 13</var>, <var data-var='time'>14:29:32</var> GMT+0 Completed -
 Maintenance has completed successfully..
Sep <var data-var='date'> 13</var>, <var data-var='time'>13:10:01</var> GMT+0 Identified -
 Maintenance is now in progress.
Sep <var data-var='date'> 13</var>, <var data-var='time'>12:53:14</var> GMT+0 Identified -
 This is a notice to let you know that planned maintenance is coming up. It’s done regularly to make sure TingleFlow services are even more reliable, fast, and safe. As a result, the TingleFlow website may experience some downtime during this timeframe..

TingleFlow Website Maintenance

Type: Incident
 Duration: 2 hours and 23 minutes
 Affected Components: 
 Aug <var data-var='date'> 20</var>, <var data-var='time'>10:21:05</var> GMT+0 Resolved -
 API is fixed..
Aug <var data-var='date'> 20</var>, <var data-var='time'>08:30:52</var> GMT+0 Monitoring -
 ### All regions except for places with server delays are now back online.

### 

North America

Central America

South America (Excludes all servers in Rio de Janeiro (BR) and Bogotá (CO) due to server delay, expected return is 20 minutes, users may be routed through servers in São Paulo (BR) or Buenos Aires (AR) or any other smaller servers that can handle extra traffic.

Europe

Africa &amp; Middle East

Asia (excluding Southeast Asia, expected return 20 minutes)

Oceania (excluding servers in Melbourne (AU) and Sydney (AU) due to server delay, expected return is 10 minutes, sers may be routed through servers in Auckland (NZ) or any other smaller servers that can handle extra traffic..
Aug <var data-var='date'> 20</var>, <var data-var='time'>08:23:02</var> GMT+0 Monitoring -
 The following regions are now back online:

North America

Central America

South America (Excludes all servers in Rio de Janeiro (BR) and Bogotá (CO) due to server delay, expected return is 20 minutes, users may be routed through servers in São Paulo (BR) or Buenos Aires (AR) or any other smaller servers that can handle extra traffic.

More regions will be returning shortly..
Aug <var data-var='date'> 20</var>, <var data-var='time'>08:11:42</var> GMT+0 Identified -
 We have identified the issue and are working on a fix. Our ETA is currently quite broad as propagation can take up to 48 hours, we do not believe it will take this long but we will keep you updated as locations come back online..
Aug <var data-var='date'> 20</var>, <var data-var='time'>07:58:13</var> GMT+0 Investigating -
 We are currently investigating this incident. Users may be unable to connect at this time..

Issues with TingleFlow.com

Type: Maintenance
 Duration: 15 minutes
 Affected Components: 
 Aug <var data-var='date'> 15</var>, <var data-var='time'>07:39:30</var> GMT+0 Completed -
 Maintenance has completed successfully..
Aug <var data-var='date'> 15</var>, <var data-var='time'>07:20:01</var> GMT+0 Identified -
 Maintenance is now in progress.
Aug <var data-var='date'> 15</var>, <var data-var='time'>07:15:53</var> GMT+0 Identified -
 We&#039;re occasionally undergo maintenance to make sure TingleFlow is constantly up to date and secure..

We're undergoing maintenance shortly.

Type: Incident
 Duration: 3 hours and 1 minute
 Affected Components: 
 Aug <var data-var='date'> 15</var>, <var data-var='time'>05:33:19</var> GMT+0 Resolved -
 This incident has been resolved..
Aug <var data-var='date'> 15</var>, <var data-var='time'>04:47:45</var> GMT+0 Monitoring -
 System appears to be operating as expected..
Aug <var data-var='date'> 15</var>, <var data-var='time'>02:32:26</var> GMT+0 Investigating -
 We are currently investigating this incident..

Some aspects of TingleFlow may be degraded

Type: Incident
 Duration: 22 minutes
 Affected Components: 
 Aug <var data-var='date'> 14</var>, <var data-var='time'>12:20:25</var> GMT+0 Resolved -
 This incident has been resolved..
Aug <var data-var='date'> 14</var>, <var data-var='time'>12:15:07</var> GMT+0 Monitoring -
 Traffic is now returning to normal..
Aug <var data-var='date'> 14</var>, <var data-var='time'>12:12:23</var> GMT+0 Monitoring -
 Help center and contacts are now back up. Website should return within the next 15 minutes..
Aug <var data-var='date'> 14</var>, <var data-var='time'>12:10:02</var> GMT+0 Identified -
 We are seeing the system stabilizing and will begin letting traffic in incrementally shortly. Our customer service desk is also returning, if you have opened a ticket, please re-send it to us when this page states everything is operational..
Aug <var data-var='date'> 14</var>, <var data-var='time'>11:58:48</var> GMT+0 Investigating -
 Users may experience issues connecting to TingleFlow. Access is currently limited due to a system bottleneck. We are investigating the issue and working to resolve it..

Internal service issue

Type: Incident
 Duration: 28 minutes
 Affected Components: 
 Aug <var data-var='date'> 14</var>, <var data-var='time'>01:30:00</var> GMT+0 Resolved -
 This incident has been resolved..
Aug <var data-var='date'> 14</var>, <var data-var='time'>01:25:27</var> GMT+0 Monitoring -
 We reroute traffic and see improvement. some user may have access again.
Aug <var data-var='date'> 14</var>, <var data-var='time'>01:17:25</var> GMT+0 Identified -
 We identify the root cause as third party issue and we reach out to them to get updates.
Aug <var data-var='date'> 14</var>, <var data-var='time'>01:01:47</var> GMT+0 Investigating -
 Multiple systems are impacted. stay tuned..

DNS/Website issues

Type: Incident
 Duration: 11 minutes
 Affected Components: 
 Aug <var data-var='date'> 3</var>, <var data-var='time'>13:00:02</var> GMT+0 Resolved -
 This incident has been resolved..
Aug <var data-var='date'> 3</var>, <var data-var='time'>12:56:47</var> GMT+0 Monitoring -
 Users are now being routed through AU-823 again. We will continue monitoring the system closely to ensure everything remains stable and is operating as expected..
Aug <var data-var='date'> 3</var>, <var data-var='time'>12:48:47</var> GMT+0 Identified -
 We are performing an emergency maintenance on AU server 823, located in the Sydney, Australia. Users routed through there could be temporarily inaccessible or might load slower than usual. Our team is currently working to restore normal performance levels. We apologize for any inconvenience caused..

Emergency Server Maintenance - AU-823

Type: Incident
 Duration: 2 days, 22 hours and 48 minutes
 Affected Components: 
 Jul <var data-var='date'> 24</var>, <var data-var='time'>10:15:22</var> GMT+0 Resolved -
 All users now have access to TingleFlow, and our systems remain stable.

We sincerely apologize for the length of this outage. It was a highly unusual incident due to both its complexity and the nature of the underlying issue. Over the coming days, we will conduct a thorough review to understand exactly what went wrong, identify areas for improvement, and help prevent similar incidents from happening again..
Jul <var data-var='date'> 24</var>, <var data-var='time'>05:06:15</var> GMT+0 Monitoring -
 We have begun allowing traffic back into TingleFlow incrementally.

Some users will regain access before others as we carefully monitor system stability and performance.

Users may regain access in the following priority order:

1. TingleFlow+ Subscribers (Creators)
2. TingleFlow+ Subscribers (Viewers)
3. Creators (users with two or more uploaded videos)
4. Viewers

To further reduce system load, access will also be rolled out gradually by geographic region. This means users within the same category may regain access at different times.

Regional rollout order:

1. North America
2. South America
3. Africa
4. Europe
5. Asia
6. Oceania

This phased rollout helps us closely monitor system performance and ensure the platform remains stable as users regain access. We appreciate everyone&#039;s patience while we complete the recovery..
Jul <var data-var='date'> 23</var>, <var data-var='time'>23:25:23</var> GMT+0 Monitoring -
 It appears that the fix has been successful, and our systems are returning to normal. We expect to begin allowing users back onto TingleFlow incrementally within the next few hours. To help ensure a stable recovery and avoid overloading our infrastructure, we will gradually restore access using DNS routing. We will share more information on which users will gain access first when we&#039;re ready to start letting users access TingleFlow again..
Jul <var data-var='date'> 23</var>, <var data-var='time'>11:20:31</var> GMT+0 Monitoring -
 We believe that we have deployed an appropriate fix. We will ensure it is working as expected, then we will be able to let traffic in incrementally if all goes well. .
Jul <var data-var='date'> 23</var>, <var data-var='time'>03:59:43</var> GMT+0 Identified -
 We have identified the exact root cause of the issue and have began working on a solution. Our team is actively working to restore all services, and we&#039;ll continue to provide updates throughout the day..
Jul <var data-var='date'> 23</var>, <var data-var='time'>00:47:11</var> GMT+0 Identified -
 We are continuing to work on a fix for this incident. .
Jul <var data-var='date'> 22</var>, <var data-var='time'>12:28:42</var> GMT+0 Identified -
 We **believe** we have identified the internal issue that contributed to the TingleFlow outage. We are currently carrying out the required fixes, testing, and maintenance work to safely restore services. Thank you for your continued patience and understanding while we work to bring TingleFlow back online..
Jul <var data-var='date'> 22</var>, <var data-var='time'>11:10:30</var> GMT+0 Investigating -
 ## TingleFlow has released a statement regarding this outage. You can read it here: &lt;https://blog.tingleflow.com/tingleflow-addresses-unplanned-service-outage/&gt;.
Jul <var data-var='date'> 22</var>, <var data-var='time'>09:12:25</var> GMT+0 Investigating -
 We believe we have narrowed down the issue and are very close to identifying it..
Jul <var data-var='date'> 22</var>, <var data-var='time'>07:25:06</var> GMT+0 Investigating -
 We are continuing to investigate the root cause of this outage. We will provide another update when available. We believe we are close to identifying the root cause..
Jul <var data-var='date'> 22</var>, <var data-var='time'>05:15:52</var> GMT+0 Investigating -
 We are continuing to investigate the root cause of this outage. We will provide another update when available..
Jul <var data-var='date'> 22</var>, <var data-var='time'>04:04:01</var> GMT+0 Investigating -
 The engineering team has dug deeper into debug logs and operating system-level metrics to better understand the issue. This data showed that Consul KV writes were getting blocked for long periods of time, resulting in what is known as “contention.”

The cause of the contention is not currently obvious, though one theory is that the shift from 64 to 128 CPU Core servers early in the outage may have made the problem worse. After reviewing the HTOP data and performance debugging data shown in the screenshots below, we have concluded that it is ideal to return to 64 Core servers, similar to those used before the outage.

We have begun preparing the hardware for this transition. Consul is installed, operating system configurations have been triple checked, and the machines have been readied for service in as detailed a manner as possible. We will then transition the Consul cluster back to 64 CPU Core servers.

This will be our fourth attempt at diagnosing the root cause of the incident..
Jul <var data-var='date'> 22</var>, <var data-var='time'>02:17:14</var> GMT+0 Investigating -
 We are continuing to look for the root cause of this issue. We deeply apologise for the inconvenience caused during this incident..
Jul <var data-var='date'> 22</var>, <var data-var='time'>00:51:55</var> GMT+0 Investigating -
 At this point, it is now clear to us that overall Consul usage is not the only contributing factor to the performance degradation. Given this realization, we will again pivot. Instead of looking at Consul from the perspective of the TingleFlow services that depend on it, we will start looking at Consul internals for clues..
Jul <var data-var='date'> 21</var>, <var data-var='time'>23:47:31</var> GMT+0 Investigating -
 We are currently looking into whether hardware failure may be a part of the issue. However, faster hardware hasn&#039;t helped and, as we now know, likely hurt stability. Resetting Consul’s internal state hasn&#039;t helped either. There is no user traffic coming in, yet Consul is still slow. We have again leveraged iptables to let traffic back into the cluster slowly. We are looking into whether the cluster simply getting pushed back into an unhealthy state by the sheer volume of thousands of containers trying to reconnect has caused this. This was our third attempt at diagnosing the root cause of the incident**, no cause has yet been identified.** 

### **We have ruled out:**

* DDoS/Cyber attack
* Frontend issues
* Authentication issues

Our next step is to reduce Consul usage and then systematically reintroduce it. 

**_TingleFlow also currently reports 0 users online, for the first time._**.
Jul <var data-var='date'> 21</var>, <var data-var='time'>22:50:02</var> GMT+0 Investigating -
 The reset went smoothly, and initially, the metrics looked good. When we removed the iptables block, the service discovery and health check load from the internal services returned as expected. However, Consul performance began to degrade again, and now we are back to where we started: 50th percentile on KV write operations was back at 4 seconds. Services and systems that depended on Consul are now starting to mark themselves “unhealthy.” The system has fell back into the problematic state. There is clearly something about our load on Consul that is causing problems, and over 12 hours into the incident, we still are unsure as to what it is..
Jul <var data-var='date'> 21</var>, <var data-var='time'>21:34:56</var> GMT+0 Investigating -
 We expected that restoring from a snapshot taken when the system was healthy would bring the cluster into a healthy state, but we had one additional concern. Even though TingleFlow did not have any user-generated traffic flowing through the system at this point, internal TingleFlow services and systems were still live and reaching out to Consul to learn the location of their dependencies and to update their health information. These reads and writes were generating a significant load on the cluster. We were worried that this load might immediately push the cluster back into an unhealthy state even if the cluster reset was successful. To address this concern, we configured iptables on the cluster to block access. This would allow us to bring the cluster back up in a controlled way and help us understand if the load we were putting on Consul independent of user traffic was part of the problem..
Jul <var data-var='date'> 21</var>, <var data-var='time'>12:20:41</var> GMT+0 Investigating -
 Based on the severity of this incident, we have changed the status to Critical. We cannot yet provide an ETA but we encourage users to stay informed here..
Jul <var data-var='date'> 21</var>, <var data-var='time'>11:27:05</var> GMT+0 Investigating -
 We are currently investigating this incident. We will provide updates when they become available..

Issues with accessing TingleFlow

Type: Incident
 Duration: 1 hour and 3 minutes
 Affected Components: 
 Jul <var data-var='date'> 20</var>, <var data-var='time'>00:27:49</var> GMT+0 Resolved -
 This incident has been resolved..
Jul <var data-var='date'> 20</var>, <var data-var='time'>00:05:56</var> GMT+0 Monitoring -
 We implemented a fix and are currently monitoring the result..
Jul <var data-var='date'> 19</var>, <var data-var='time'>23:59:54</var> GMT+0 Identified -
 We believe we have identified the cause of this issue and we will begin working on a fix..
Jul <var data-var='date'> 19</var>, <var data-var='time'>23:24:41</var> GMT+0 Investigating -
 We are currently investigating this incident..

Website Issues

Type: Incident
 Duration: 6 hours and 25 minutes
 Affected Components: 
 Jun <var data-var='date'> 1</var>, <var data-var='time'>07:12:00</var> GMT+0 Resolved -
 This incident has been resolved..
Jun <var data-var='date'> 1</var>, <var data-var='time'>01:49:50</var> GMT+0 Monitoring -
 We implemented a fix and are currently monitoring the result..
Jun <var data-var='date'> 1</var>, <var data-var='time'>01:38:03</var> GMT+0 Investigating -
 We are currently investigating this incident..

Mobile users may see loading errors

Type: Maintenance
 Duration: 8 minutes
 Affected Components: 
 May <var data-var='date'> 30</var>, <var data-var='time'>11:22:39</var> GMT+0 Completed -
 Maintenance has completed successfully..
May <var data-var='date'> 30</var>, <var data-var='time'>11:15:01</var> GMT+0 Identified -
 Maintenance is now in progress.
May <var data-var='date'> 30</var>, <var data-var='time'>11:15:00</var> GMT+0 Identified -
 We are planning for a scheduled maintenance during that time. Some services may be unavailable..

Planned Website Maintenance

Type: Incident
 Duration: 12 hours and 21 minutes
 Affected Components: 
 Apr <var data-var='date'> 4</var>, <var data-var='time'>00:43:10</var> GMT+0 Resolved -
 This incident has been resolved..
Apr <var data-var='date'> 3</var>, <var data-var='time'>12:22:38</var> GMT+0 Investigating -
 We are currently investigating this incident..

Issues with updating profile pictures

Type: Maintenance
 Duration: 3 hours
 Affected Components: 
 Mar <var data-var='date'> 26</var>, <var data-var='time'>14:00:00</var> GMT+0 Completed -
 Maintenance has completed successfully.
Mar <var data-var='date'> 26</var>, <var data-var='time'>11:00:01</var> GMT+0 Identified -
 Maintenance is now in progress.
Mar <var data-var='date'> 26</var>, <var data-var='time'>11:00:00</var> GMT+0 Identified -
 This is a notice to let you know that planned maintenance is coming up. It’s done regularly to make sure TingleFlow services are even more reliable, fast, and safe. As a result, the TingleFlow website may experience some downtime during this timeframe..

Planned Server Maintenance

Type: Incident
 Duration: 7 hours and 14 minutes
 Affected Components: 
 Mar <var data-var='date'> 23</var>, <var data-var='time'>21:40:51</var> GMT+0 Resolved -
 This incident has been resolved..
Mar <var data-var='date'> 23</var>, <var data-var='time'>14:26:21</var> GMT+0 Investigating -
 We are currently investigating this incident..

Some users may have issues loading user profiles.

Type: Incident
 Duration: 26 minutes
 Affected Components: 
 Mar <var data-var='date'> 23</var>, <var data-var='time'>11:52:44</var> GMT+0 Resolved -
 This incident has been resolved..
Mar <var data-var='date'> 23</var>, <var data-var='time'>11:41:13</var> GMT+0 Monitoring -
 We implemented a fix and are currently monitoring the result..
Mar <var data-var='date'> 23</var>, <var data-var='time'>11:31:57</var> GMT+0 Identified -
 Identified as data streaming issue..
Mar <var data-var='date'> 23</var>, <var data-var='time'>11:27:00</var> GMT+0 Investigating -
 We are currently investigating this incident..

Issues loading TingleFlow

Type: Maintenance
 Duration: 30 minutes
 Affected Components: 
 Mar <var data-var='date'> 21</var>, <var data-var='time'>14:00:00</var> GMT+0 Completed -
 Maintenance has completed successfully.
Mar <var data-var='date'> 21</var>, <var data-var='time'>13:30:01</var> GMT+0 Identified -
 Maintenance is now in progress.
Mar <var data-var='date'> 21</var>, <var data-var='time'>13:30:00</var> GMT+0 Identified -
 We are planning for a scheduled maintenance during this time..

Website maintenance

Type: Incident
 Duration: 10 hours and 53 minutes
 Affected Components: 
 Mar <var data-var='date'> 18</var>, <var data-var='time'>23:28:19</var> GMT+0 Resolved -
 This incident has been resolved..
Mar <var data-var='date'> 18</var>, <var data-var='time'>13:59:26</var> GMT+0 Monitoring -
 The issues have resided and systems are operating nominally or in clear recovery. We will continue to monitor. We are letting users back in incrementally..
Mar <var data-var='date'> 18</var>, <var data-var='time'>12:35:48</var> GMT+0 Investigating -
 We are aware of the issues impacting our services and are working on mitigations..

Type: Incident
 Duration: 5 hours and 52 minutes
 Affected Components: 
 Mar <var data-var='date'> 14</var>, <var data-var='time'>07:31:41</var> GMT+0 Resolved -
 This incident has been resolved..
Mar <var data-var='date'> 14</var>, <var data-var='time'>01:39:18</var> GMT+0 Investigating -
 We are currently investigating this incident. **A small number of videos may not play.**.

Loading issues on a small subset of videos

Type: Incident
 Duration: 13 days, 22 hours and 8 minutes
 Affected Components: 
 Mar <var data-var='date'> 25</var>, <var data-var='time'>09:14:37</var> GMT+0 Resolved -
 This incident has been resolved..
Mar <var data-var='date'> 11</var>, <var data-var='time'>11:06:31</var> GMT+0 Investigating -
 We are currently investigating this incident..

Issues with AI support

Type: Maintenance
 Duration: 2 hours
 Affected Components: 
 Mar <var data-var='date'> 9</var>, <var data-var='time'>03:00:00</var> GMT+0 Completed -
 Maintenance has completed successfully.
Mar <var data-var='date'> 9</var>, <var data-var='time'>01:00:01</var> GMT+0 Identified -
 Maintenance is now in progress.
Mar <var data-var='date'> 9</var>, <var data-var='time'>01:00:00</var> GMT+0 Identified -
 We are planning for a scheduled maintenance during that time. Access to TingleFlow may be slow or disrupted during this time. This will affect users that connect to TingleFlow via the Singapore region..

Server WEB2190 Maintenance

Type: Incident
 Duration: 3 days, 13 hours and 10 minutes
 Affected Components: 
 Mar <var data-var='date'> 8</var>, <var data-var='time'>01:37:50</var> GMT+0 Resolved -
 This incident has been resolved..
Mar <var data-var='date'> 5</var>, <var data-var='time'>08:54:25</var> GMT+0 Monitoring -
 We implemented a fix and are currently monitoring the result..
Mar <var data-var='date'> 4</var>, <var data-var='time'>12:27:22</var> GMT+0 Investigating -
 We are currently investigating this incident..

Potential issues with 2FA/MFA enrolment and verification

Type: Maintenance
 Duration: 10 minutes
 Affected Components: 
 Dec <var data-var='date'> 30</var>, <var data-var='time'>23:24:56</var> GMT+0 Completed -
 Maintenance has completed successfully..
Dec <var data-var='date'> 30</var>, <var data-var='time'>23:15:01</var> GMT+0 Identified -
 Maintenance is now in progress.
Dec <var data-var='date'> 30</var>, <var data-var='time'>23:15:00</var> GMT+0 Identified -
 We are planning for a scheduled maintenance during that time..

Type: Maintenance
 Duration: 26 days, 2 hours and 48 minutes
 Affected Components: 
 Oct <var data-var='date'> 29</var>, <var data-var='time'>21:10:00</var> GMT+0 Completed -
 Maintenance has completed successfully..
Oct <var data-var='date'> 29</var>, <var data-var='time'>21:00:00</var> GMT+0 Identified -
 You may experience intermittent issues while maintenance is taking place..

Is TingleFlow down?

TingleFlow status is UP

Lastest published events

TingleFlow down for upgrades

TingleFlow Radio and Sparks unavailable for some users

Direct Messaging CDN Migration

TingleFlow Services, Components and Regions

Recent TingleFlow events

TingleFlow down for upgrades

TingleFlow Radio and Sparks unavailable for some users

Direct Messaging CDN Migration

Content Delivery Migration

TingleFlow Accessibility Issues in Europe

Email delivery issues