Aug <var data-var='date'>24</var>, <var data-var='time'>13:00</var> EDT Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>24</var>, <var data-var='time'>10:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>19</var>, <var data-var='time'>10:48</var> EDT Scheduled - We are doing a scheduled maintenance on our search infrastructure. During this time, search may not work for your team. Once the migration is complete, everything should be back to normal.

Maintenance on Search Infrastructure

Jun <var data-var='date'>10</var>, <var data-var='time'>22:52</var> EDT Resolved - We have been able to identify the issue caused by heroku, and make a work around to get tettra accessible once again Jun <var data-var='date'>10</var>, <var data-var='time'>22:34</var> EDT Identified - We are experience server issues with our hosting service Heroku (Salesforce). We are actively investigating the issue, and will update as soon as we have identified the issue. Customers will have issues logging in and using Tettra.

Heroku server issues

Jun <var data-var='date'>10</var>, <var data-var='time'>18:42</var> EDT Resolved - The app is up and running, with degraded performance while Heroku fully fixes the issue. Tettra will be accessible, but expect some slower response times until the issue is fully resolve Jun <var data-var='date'>10</var>, <var data-var='time'>14:46</var> EDT Update - Heroku has identified the likely cause of the issue and is working on a solution. We’ll provide an update when additional information becomes available. Jun <var data-var='date'>10</var>, <var data-var='time'>12:00</var> EDT Update - We're continuing to work on a fix. We have no other update right now. We’ll provide another update in 30 minutes or sooner if additional information becomes available. Jun <var data-var='date'>10</var>, <var data-var='time'>10:13</var> EDT Update - We're continuing to investigate the Heroku outage and are looking for a solution. Our web app continues to be unreachable. We’ll provide an update in 30 minutes or sooner if additional information becomes available. Jun <var data-var='date'>10</var>, <var data-var='time'>07:22</var> EDT Update - We’ve confirmed that the outage is linked to a broader Salesforce incident that began at approximately 06:17 UTC and is impacting multiple services, including Heroku—our infrastructure provider. Salesforce has identified ongoing issues with multi-factor authentication (MFA) across several of their clouds. While Heroku is affected, the root cause remains under investigation. They are coordinating with a third-party cloud provider and reviewing recent changes to the Heroku environment. As a result, customers will continue to be unable to access app.tettra.co until Salesforce resolves these issues. We are actively monitoring the situation and will update customers as soon as we have more information. Jun <var data-var='date'>10</var>, <var data-var='time'>03:46</var> EDT Identified - We are currently investigating a major outage impacting our application. This is due to an upstream infrastructure failure with our hosting provider, Heroku. Users may be unable to access any part of the application at this time. We’ve reached out to Heroku support and are closely monitoring their incident updates.

Full Outage

Feb <var data-var='date'>19</var>, <var data-var='time'>13:21</var> EST Resolved - We have identified and fixed the issue. We are currently monitoring for on-going problems. Feb <var data-var='date'>19</var>, <var data-var='time'>12:14</var> EST Update - We are continuing to investigate this issue. Feb <var data-var='date'>19</var>, <var data-var='time'>11:37</var> EST Investigating - We are currently investigating an issue casuing a partial outage to the Tettra app. The Tettra app itself still loads, but pages are not loading properly some of the time.

Partial Outage - Pages Not Loading Properly [Investigating]

Apr <var data-var='date'>19</var>, <var data-var='time'>11:16</var> EDT Resolved - We have restored access to all the affected accounts and are monitoring the situation. Apr <var data-var='date'>19</var>, <var data-var='time'>10:14</var> EDT Identified - We have identified the issue with our billing system that is blocking some paid teams from accessing their customers. We are currently in the process of fixing it. Apr <var data-var='date'>19</var>, <var data-var='time'>10:13</var> EDT Investigating - We are investigating an issue with our billing system that is defaulting some paid teams to the billing page.

Partial Outage - Some paid accounts defaulting to billing screen

Feb <var data-var='date'>14</var>, <var data-var='time'>00:19</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>13</var>, <var data-var='time'>18:04</var> EST Identified - The issue has been identified and a fix is being implemented.

Search not working for a small % of teams

Feb <var data-var='date'> 3</var>, <var data-var='time'>22:10</var> EST Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 3</var>, <var data-var='time'>21:00</var> EST Scheduled - We will be undergoing scheduled maintenance during this time.

Database upgrades

Dec <var data-var='date'>11</var>, <var data-var='time'>11:17</var> EST Resolved - This incident has been resolved. Dec <var data-var='date'>11</var>, <var data-var='time'>11:17</var> EST Update - A fix for this issue has been deployed. Dec <var data-var='date'>11</var>, <var data-var='time'>10:37</var> EST Identified - Read-only users are not able to view pages or the sidebar right now. We're working on rolling back the deploy the caused this issue.

Pages and sidebar not loading for read-only users

Jul <var data-var='date'>13</var>, <var data-var='time'>22:58</var> EDT Resolved - This incident has been resolved.

Some users unable to create new team

May <var data-var='date'>25</var>, <var data-var='time'>18:00</var> EDT Resolved - We were having issues with our SSO provider integration. This has now been resolved.

Intermittent issues with SSO Login

Aug <var data-var='date'>23</var>, <var data-var='time'>20:38</var> EDT Resolved - A workaround for the DNS issue has been implemented and the app is accessible as per normal Aug <var data-var='date'>23</var>, <var data-var='time'>20:03</var> EDT Identified - Our hosting provider Heroku is currently experiencing DNS issues, which is affecting the Tettra app’s ability to properly load. We are currently working to identify a potential workaround. https://status.heroku.com/incidents/2453

Tettra web app is down

Feb <var data-var='date'>24</var>, <var data-var='time'>13:03</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>24</var>, <var data-var='time'>12:19</var> EST Update - We are continuing to monitor for any further issues. Feb <var data-var='date'>24</var>, <var data-var='time'>12:18</var> EST Monitoring - Connectivity to the webapp is recovering. Feb <var data-var='date'>24</var>, <var data-var='time'>11:45</var> EST Identified - Our hosting provider, Heroku, is experiencing an outage. Feb <var data-var='date'>24</var>, <var data-var='time'>11:41</var> EST Investigating - The Tettra webapp is not responding to requests. We are investigating the issue.

Webapp is down

Feb <var data-var='date'>21</var>, <var data-var='time'>15:46</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>21</var>, <var data-var='time'>14:05</var> EST Monitoring - Our transactional email service provider, Mailgun, is experiencing API issues causing our email delivery to fail. We are monitoring the situation. https://status.mailgun.com/incidents/0hgvxpzzhvcq

Email delivery delayed due to service provider errors

Jan <var data-var='date'> 4</var>, <var data-var='time'>14:49</var> EST Resolved - We believe slack's outage is now contained to parts of Slack that do not impact Tettra. Please let us know if you see any further issues with the Slack integration. Jan <var data-var='date'> 4</var>, <var data-var='time'>13:31</var> EST Monitoring - Slack is starting to come back online, so we hope all issues with our integration will resolve soon. Jan <var data-var='date'> 4</var>, <var data-var='time'>11:24</var> EST Update - We are continuing to monitor Slack. You can see their own status page, here: https://status.slack.com/ Jan <var data-var='date'> 4</var>, <var data-var='time'>10:52</var> EST Investigating - Slack is currently experiencing system-wide issues. Tettra users may be unable to sign in using Slack or use the Slack integration within Tettra. We are monitoring the situation in case there is anything we can do to mitigate these issues with Slack.

Slack Instability

Dec <var data-var='date'>26</var>, <var data-var='time'>15:54</var> EST Resolved - Email delivery and links are operational again. The 'share' page via email functionality has been temporarily disabled while we mitigate further spam attempts. Dec <var data-var='date'>26</var>, <var data-var='time'>12:34</var> EST Update - Links in existing/delivered emails are also disabled while we continue to work on a fix. Dec <var data-var='date'>26</var>, <var data-var='time'>11:10</var> EST Identified - All emails from our application have been suspended since Dec 24 approximately 22:24 UTC due to a spam attack. This includes all automated notification emails, invitation emails, and page share emails. We have identified and suspended the malicious account(s) and are in the process of re-enabling emails.

Email delivery disabled due to spam attack

May <var data-var='date'>17</var>, <var data-var='time'>13:36</var> EDT Resolved - Content embedding functionality has been restored. Thanks for your patience. May <var data-var='date'>16</var>, <var data-var='time'>08:22</var> EDT Monitoring - Embedly is still working on a fix. At this time there is no ETA as to when content embedding will be available again. We will update here as we know more. Original incident: http://status.embed.ly/incidents/tdkr78g2qq5x May <var data-var='date'>15</var>, <var data-var='time'>16:03</var> EDT Identified - We've identified an issue with our content embedding provider that is causing content embedding in pages to fail. We'll update this issue once the problem is resolved.

Content embeds are failing

Apr <var data-var='date'>17</var>, <var data-var='time'>17:00</var> EDT Resolved - From approximately Friday 4/17 5pm EST to Saturday 11am EST, readonly users were not able to access Tettra pages on the webapp.

Readonly users unable to access Tettra pages

Mar <var data-var='date'>16</var>, <var data-var='time'>17:37</var> EDT Resolved - Access to the webapp has returned to normal. We will continue to keep an eye on the underlying Heroku incident that caused this and the outage earlier today: https://status.heroku.com/incidents/1974 Mar <var data-var='date'>16</var>, <var data-var='time'>16:01</var> EDT Monitoring - We are continuing to monitor the app as it recovers. Mar <var data-var='date'>16</var>, <var data-var='time'>15:51</var> EDT Investigating - app.tettra.co is unavailable due to problems with our service provider. We are investigating the issue.

Webapp unavailable

Mar <var data-var='date'>16</var>, <var data-var='time'>13:36</var> EDT Resolved - This incident has been resolved. Mar <var data-var='date'>16</var>, <var data-var='time'>12:31</var> EDT Update - We are continuing to monitor for any further issues. Mar <var data-var='date'>16</var>, <var data-var='time'>12:30</var> EDT Monitoring - The app is loading normally again. We'll keep monitoring until we hear from our hosting provider. Mar <var data-var='date'>16</var>, <var data-var='time'>12:25</var> EDT Update - Our service provider Heroku is experiencing widespread platform issues. We will update as we have more information. Mar <var data-var='date'>16</var>, <var data-var='time'>12:21</var> EDT Investigating - Visitors to app.tettra.co are seeing a Cloudflare error. We are investigating the cause.

Web app unavailable

Nov <var data-var='date'> 2</var>, <var data-var='time'>20:26</var> EDT Completed - The scheduled maintenance has been completed. Nov <var data-var='date'> 2</var>, <var data-var='time'>20:25</var> EDT Update - Scheduled maintenance is still in progress. We will provide updates as necessary. Nov <var data-var='date'> 2</var>, <var data-var='time'>20:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>28</var>, <var data-var='time'>15:59</var> EDT Scheduled - We will be updating our SSL certificates on our primary database during this time, which requires a reboot. We expect minimal service disruption but you may notice a few minutes of slower response times.

Database SSL Certificate Update

Oct <var data-var='date'>26</var>, <var data-var='time'>20:45</var> EDT Completed - The scheduled maintenance has been completed. Oct <var data-var='date'>26</var>, <var data-var='time'>20:15</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>17</var>, <var data-var='time'>14:19</var> EDT Scheduled - We will be updating our SSL certificates on our primary database during this time, which requires a reboot. Expected downtime is less than 5 minutes.

Sep <var data-var='date'> 5</var>, <var data-var='time'>11:47</var> EDT Resolved - We have successfully mitigated the connection issues, and will work to assess what we can do to prevent such issues in the future. Sep <var data-var='date'> 5</var>, <var data-var='time'>11:31</var> EDT Update - We have narrowed down the issue to problems connecting to where we store data. We have contacted our service provider's support team but are also further investigating ourselves. Sep <var data-var='date'> 5</var>, <var data-var='time'>11:10</var> EDT Investigating - We are getting alerts of degraded performance to the point where some users are unable to view their content or log in. We are investigating.

Slowness and Instability

Jul <var data-var='date'>10</var>, <var data-var='time'>20:28</var> EDT Resolved - Our billing provider has recovered. Please let us know if you see any more 500 errors. Jul <var data-var='date'>10</var>, <var data-var='time'>17:40</var> EDT Identified - We're aware of intermittent 500 errors being caused by issues with our billing provider. We will update you again when the problems have been resolved.

Intermittent 500 errors

Jul <var data-var='date'> 2</var>, <var data-var='time'>10:19</var> EDT Resolved - Cloudflare has implemented a fix for the network issue and is currently monitoring the results. The application is available again. We are continuing to monitor the situation. Jul <var data-var='date'> 2</var>, <var data-var='time'>10:17</var> EDT Update - We are continuing to investigate this issue. Jul <var data-var='date'> 2</var>, <var data-var='time'>10:11</var> EDT Investigating - We are currently investigating an issue with our DNS provider Cloudflare.

Web application unavailable

Feb <var data-var='date'> 9</var>, <var data-var='time'>19:01</var> EST Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 9</var>, <var data-var='time'>18:00</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'> 7</var>, <var data-var='time'>13:54</var> EST Update - We will be undergoing scheduled maintenance during this time. Feb <var data-var='date'> 7</var>, <var data-var='time'>13:51</var> EST Scheduled - Between 6-7pm EST on 02/09/2019, we will be performing a scheduled maintenance task. Expected downtime is <5 minutes. All users will be prompted to log back in once the task is complete.

Is Tettra down?

Tettra status is UP

Lastest published events

Maintenance on Search Infrastructure

Heroku server issues

Full Outage

Tettra Services, Components and Regions

Recent Tettra events

Maintenance on Search Infrastructure

Heroku server issues

Full Outage

Partial Outage - Pages Not Loading Properly [Investigating]

Partial Outage - Some paid accounts defaulting to billing screen

Search not working for a small % of teams