Jan <var data-var='date'> 7</var>, <var data-var='time'>22:06</var> UTC Resolved - Response times are back to normal. We are continuing to monitor performance. Jan <var data-var='date'> 7</var>, <var data-var='time'>21:51</var> UTC Investigating - We're seeing a high P95 response times for requests to our contact endpoints. We are investigating the issue and will update as we know more.

Slow Requests

Jun <var data-var='date'>12</var>, <var data-var='time'>19:32</var> UTC Resolved - Cloudflare is still experiencing an outage, but we have mitigated the issue on our end so that there is no user impact while they resolve. Please contact support if you experience issues connecting to calls. Jun <var data-var='date'>12</var>, <var data-var='time'>18:52</var> UTC Identified - Cloudflare is continuing to update their outage status: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp Jun <var data-var='date'>12</var>, <var data-var='time'>18:33</var> UTC Investigating - TURN connections are currently impacted by Cloudflare's outage: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp We are investigating temporary mitigations and monitoring the Cloudflare incident status.

Interimittent TURN errors

Aug <var data-var='date'>10</var>, <var data-var='time'>10:33</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>10</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'> 1</var>, <var data-var='time'>15:07</var> UTC Scheduled - Tuple will be down for scheduled maintenance on August 10th, 2024 at 6:00am EST. We're anticipating between 30 and 60 minutes of downtime. During this time, Tuple's web dashboard (production.tuple.app) will be unavailable, and your desktop client will show a "Loading" state.

Saturday 10th August, 2024

Apr <var data-var='date'>29</var>, <var data-var='time'>18:17</var> UTC Resolved - This incident has been resolved. Please get in touch with support@tuple.app if you experience any issues. Apr <var data-var='date'>29</var>, <var data-var='time'>18:09</var> UTC Monitoring - We've implemented a fix and are monitoring the results. Apr <var data-var='date'>29</var>, <var data-var='time'>17:46</var> UTC Investigating - We are currently investigating degraded performance across the web dashboard and native applications due to an underlying issue with our database infrastructure.

Database performance issues

Apr <var data-var='date'>19</var>, <var data-var='time'>19:40</var> UTC Resolved - Crash reports have ceased. We're considering this incident fully resolved! Apr <var data-var='date'>19</var>, <var data-var='time'>19:33</var> UTC Monitoring - We've reverted to the previous release. If you're experiencing issues, please update your app. We'll resolve this once we're confident no further crashes are occurring. Apr <var data-var='date'>19</var>, <var data-var='time'>19:02</var> UTC Identified - We're seeing issues with our latest release, deployed today, that's causing a subset of users to experience app crashes while calling. We're reverting the release to get things back to normal; that should be ready and deployed in the next 15 minutes.

App Crash with Latest Release

Mar <var data-var='date'> 6</var>, <var data-var='time'>22:49</var> UTC Resolved - All client response times are back to normal and we're considering this incident resolved. Please get in touch with support@tuple.app if you experience any issues. Mar <var data-var='date'> 6</var>, <var data-var='time'>22:42</var> UTC Monitoring - A fix has been implemented for affected teams, and response times for the affected API route are back to normal. Mar <var data-var='date'> 6</var>, <var data-var='time'>22:18</var> UTC Identified - We've identified a performance issue with our Call URLs API route that is affecting a small number of teams. We've put a temporary fix in place for those teams that will restore application access with some diminished functionality around Call URLs, and the team is actively working on a permanent fix. The majority of Tuple users remain unaffected. Mar <var data-var='date'> 6</var>, <var data-var='time'>21:53</var> UTC Investigating - We are seeing failed requests to our call URLs endpoint for specific teams. Not all users are affected, and we're currently investigating the issue to determine the root cause.

Web Backend Request Failures

Jan <var data-var='date'>12</var>, <var data-var='time'>21:42</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>12</var>, <var data-var='time'>21:33</var> UTC Update - Services are operational, and we're continuing to monitor our web application metrics before considering this fully resolved. Jan <var data-var='date'>12</var>, <var data-var='time'>20:56</var> UTC Monitoring - Presence service traffic was scaled to zero to allow the backend to stabilize. Things appear to be operational, but we're actively monitoring the situation to ensure the incident is fully resolved. Jan <var data-var='date'>12</var>, <var data-var='time'>20:52</var> UTC Investigating - We're experiencing a large number of requests through our presence service that are overwhelming our web backend.

Tuple Backend Outage

Sep <var data-var='date'>25</var>, <var data-var='time'>15:46</var> UTC Resolved - We're fully back online and client connections are succeeding. Sep <var data-var='date'>25</var>, <var data-var='time'>15:40</var> UTC Identified - System health is recovering, and some users are successfully reconnecting. We're increasing infrastructure capacity slowly to accommodate the backlog of reconnecting clients. Sep <var data-var='date'>25</var>, <var data-var='time'>15:17</var> UTC Investigating - We are currently investigating an issue with clients being unable to connect after a planned production deployment. Ongoing calls appear to be unaffected.

Native client connection issues

Jan <var data-var='date'>25</var>, <var data-var='time'>18:59</var> UTC Resolved - Directory status for all users is up-to-date, and the signaling server has continued to process requests successfully. Jan <var data-var='date'>25</var>, <var data-var='time'>17:48</var> UTC Update - We've seen some instances of stale presence data persisting in the directory and are monitoring the relevant queue as it is processed. Jan <var data-var='date'>25</var>, <var data-var='time'>17:36</var> UTC Monitoring - Tuple contact directories are loading. We're actively monitoring the service while clients reconnect to ensure that the issue is resolved. Jan <var data-var='date'>25</var>, <var data-var='time'>17:29</var> UTC Update - We've attempted to roll back our signaling service to a previous release, but are still experiencing issues with our load balancer correctly negotiating connections. Our team is still actively working to resolve this issue. Jan <var data-var='date'>25</var>, <var data-var='time'>17:13</var> UTC Update - We're continuing to work on a fix for this issue. Jan <var data-var='date'>25</var>, <var data-var='time'>17:01</var> UTC Identified - We've identified an issue with our signaling server that is preventing the team directory from loading. We're working to resolve the issue now.

Presence Signaling Service Issues

Dec <var data-var='date'>17</var>, <var data-var='time'>14:14</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'>17</var>, <var data-var='time'>00:00</var> UTC Monitoring - Our Email provider is experiencing an outage. Application emails are delayed.

Email Provider Outage

Feb <var data-var='date'>24</var>, <var data-var='time'>17:08</var> UTC Resolved - Service appears to have been restored. Feb <var data-var='date'>24</var>, <var data-var='time'>16:54</var> UTC Monitoring - We are currently down due to downtime at Heroku, an underlying provider of ours. You will likely be unable to sign in or make calls at this time. Their incident response can be tracked here: https://status.heroku.com/incidents/2402 (Unfortunately, their status page appears to be loading only intermittently.)

App down

Nov <var data-var='date'> 3</var>, <var data-var='time'>17:54</var> UTC Resolved - This issue has been resolved. Sorry about that folks! Nov <var data-var='date'> 3</var>, <var data-var='time'>16:06</var> UTC Monitoring - We've released a fix. Please restart Tuple to force an update to the latest version. This should resolve the issue. Nov <var data-var='date'> 3</var>, <var data-var='time'>16:03</var> UTC Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'> 3</var>, <var data-var='time'>16:01</var> UTC Identified - We believe we have identified the issue. We're building a new release and are deploying it imminently. Nov <var data-var='date'> 3</var>, <var data-var='time'>15:34</var> UTC Investigating - We're aware that users are getting stuck in a login loop/crash after this morning's update. We're investigating.

Login loop/crash after update

Sep <var data-var='date'>21</var>, <var data-var='time'>13:13</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'>21</var>, <var data-var='time'>12:33</var> UTC Investigating - Some users are reporting that they cannot log in to the Tuple client.

Is Tuple down?

Tuple status is UP

Lastest published events

Slow Requests

Interimittent TURN errors

Saturday 10th August, 2024

Tuple Services, Components and Regions

Recent Tuple events

Slow Requests

Interimittent TURN errors

Saturday 10th August, 2024

Database performance issues

App Crash with Latest Release

Web Backend Request Failures