Jun <var data-var='date'>25</var>, <var data-var='time'>13:15</var> PDT Resolved - All workflows and interfaces are back to normal. Jun <var data-var='date'>25</var>, <var data-var='time'>11:57</var> PDT Investigating - We are seeing higher error rates and slow workflow execution times in production. We are investigating the root cause.

High errors rate in workflow execution + slowness in app loading

Jun <var data-var='date'>19</var>, <var data-var='time'>02:09</var> PDT Resolved - Incident is resolve, AI Agents are back working and responding. Jun <var data-var='date'>19</var>, <var data-var='time'>02:07</var> PDT Identified - We're seeing an issue with agents not responding at all at the moment. We've identified the root cause, and are working on a fix.

AI Agents are not responding

May <var data-var='date'>24</var>, <var data-var='time'>05:00</var> PDT Completed - The scheduled maintenance has been completed. May <var data-var='date'>24</var>, <var data-var='time'>02:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>10</var>, <var data-var='time'>04:01</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Scheduled maintenance

May <var data-var='date'>15</var>, <var data-var='time'>12:26</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>15</var>, <var data-var='time'>08:41</var> PDT Update - We are continuing to monitor for any further issues. May <var data-var='date'>15</var>, <var data-var='time'>08:30</var> PDT Monitoring - A fix was implemented and we are currently monitoring the situation. May <var data-var='date'>15</var>, <var data-var='time'>08:14</var> PDT Identified - We discovered an error with Sharepoint data sources breaking due to an invalid key error. We found the root cause and are working on a fix.

Sharepoint data sources errors

Apr <var data-var='date'>26</var>, <var data-var='time'>02:22</var> PDT Completed - The scheduled maintenance has been completed. Apr <var data-var='date'>26</var>, <var data-var='time'>01:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'>19</var>, <var data-var='time'>01:02</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Apr <var data-var='date'>16</var>, <var data-var='time'>02:23</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>16</var>, <var data-var='time'>01:33</var> PDT Investigating - We're investigating a slowness in processing of Tonkean items, actions and interfaces

Slowness in processing of Tonkean items, actions and interfaces

Apr <var data-var='date'>15</var>, <var data-var='time'>14:22</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>15</var>, <var data-var='time'>14:16</var> PDT Update - We are continuing to monitor for any further issues. Apr <var data-var='date'>15</var>, <var data-var='time'>14:11</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>15</var>, <var data-var='time'>13:32</var> PDT Identified - The issue has been identified and a fix is being implemented. Apr <var data-var='date'>15</var>, <var data-var='time'>12:33</var> PDT Update - We are continuing to investigate this issue. Apr <var data-var='date'>15</var>, <var data-var='time'>10:32</var> PDT Investigating - We've seen slowness in processing of Tonkean items. We're investigating the root cause and a possible fix.

Slow processing of Tonkean items and access to the Tonkean app

Nov <var data-var='date'>25</var>, <var data-var='time'>01:09</var> PST Resolved - This incident has been resolved. Nov <var data-var='date'>25</var>, <var data-var='time'>00:51</var> PST Investigating - We are currently investigating this issue.

403 Forbidden Errors Detected on Specific Endpoints – Under Investigation

Nov <var data-var='date'> 2</var>, <var data-var='time'>03:00</var> PST Completed - The scheduled maintenance has been completed. Nov <var data-var='date'> 2</var>, <var data-var='time'>01:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>26</var>, <var data-var='time'>01:40</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Oct <var data-var='date'>20</var>, <var data-var='time'>17:33</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>04:35</var> PDT Monitoring - We are aware of an ongoing outage affecting AWS services, which may impact parts of our platform. Our team is actively monitoring the situation and assessing potential effects on our systems.

AWS Regional Outage – Monitoring Impact

Oct <var data-var='date'>16</var>, <var data-var='time'>05:31</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>16</var>, <var data-var='time'>05:18</var> PDT Investigating - We are currently investigating this issue.

Autoscaler Permission Change Incident

Sep <var data-var='date'>29</var>, <var data-var='time'>13:21</var> PDT Resolved - This incident has been resolved. Sep <var data-var='date'>29</var>, <var data-var='time'>11:53</var> PDT Investigating - We are experiencing slowness in execution of actions. We're looking into it.

Slowness in handling of Tonkean actions

Sep <var data-var='date'>28</var>, <var data-var='time'>06:00</var> PDT Completed - The scheduled maintenance has been completed. Sep <var data-var='date'>28</var>, <var data-var='time'>01:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>15</var>, <var data-var='time'>04:40</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Sep <var data-var='date'> 9</var>, <var data-var='time'>14:14</var> PDT Resolved - Workflow Runtime History events should be displayed as usual now. Sep <var data-var='date'> 9</var>, <var data-var='time'>06:18</var> PDT Investigating - We are investigating an issue causing a delay in displaying events in the History view in Tonkean. The workflows and actions still operate as expected.

Events in Workflow Runtime History are displayed in delay

Aug <var data-var='date'>27</var>, <var data-var='time'>07:30</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'>27</var>, <var data-var='time'>07:06</var> PDT Identified - The issue has been identified and a fix is being implemented.

Workspace App Pages with Tabs not loading

Aug <var data-var='date'>17</var>, <var data-var='time'>04:22</var> PDT Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>17</var>, <var data-var='time'>00:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'> 6</var>, <var data-var='time'>00:52</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Schedule maintenance

Aug <var data-var='date'> 8</var>, <var data-var='time'>03:12</var> PDT Resolved - Processing in EU environment is back to normal. Aug <var data-var='date'> 8</var>, <var data-var='time'>02:24</var> PDT Investigating - We are experience slowness in processing events in Tonkean in the EU environment. Other environments are not affected.

EU Environment - Slowness in Tonkean Processing

Aug <var data-var='date'> 5</var>, <var data-var='time'>16:46</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'> 5</var>, <var data-var='time'>16:46</var> PDT Monitoring - Fix was applied, workflows are back to normal. We're monitoring. Aug <var data-var='date'> 5</var>, <var data-var='time'>16:23</var> PDT Update - We are continuing to investigate this issue. Aug <var data-var='date'> 5</var>, <var data-var='time'>16:23</var> PDT Investigating - We are currently investigating this issue.

Tonkean Workflows Failures

Aug <var data-var='date'> 5</var>, <var data-var='time'>07:09</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'> 5</var>, <var data-var='time'>06:30</var> PDT Investigating - We are currently investigating this issue.

We’re currently investigating issues affecting some of our integration services, including delays and failures in inbound and outbound requests.

Jul <var data-var='date'>16</var>, <var data-var='time'>21:29</var> PDT Resolved - This incident has been resolved. Jul <var data-var='date'>16</var>, <var data-var='time'>14:26</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>16</var>, <var data-var='time'>14:00</var> PDT Investigating - Our team is actively investigating the issue and working to restore normal performance as quickly as possible. We will provide updates as we learn more.

We are currently experiencing slowness in one of our databases, which is impacting multiple components across the system.

Jul <var data-var='date'> 9</var>, <var data-var='time'>16:34</var> PDT Resolved - This incident has been resolved. Jul <var data-var='date'> 9</var>, <var data-var='time'>16:19</var> PDT Investigating - We are investigating an issue with Tonkean app not loading at all.

Tonkean App not responsive

May <var data-var='date'>29</var>, <var data-var='time'>15:55</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>29</var>, <var data-var='time'>15:51</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>29</var>, <var data-var='time'>15:15</var> PDT Investigating - We are currently investigating this issue.

We are experiencing an unexpected load

Apr <var data-var='date'>28</var>, <var data-var='time'>14:57</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>28</var>, <var data-var='time'>14:06</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>28</var>, <var data-var='time'>12:29</var> PDT Investigating - We are currently experiencing slowness across several services in our system. Our engineering team is actively investigating the root cause and working to restore optimal performance as quickly as possible. We apologize for the inconvenience and will provide updates as we learn more.

System Slowness Across Multiple Services

Apr <var data-var='date'>20</var>, <var data-var='time'>03:12</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>20</var>, <var data-var='time'>03:12</var> PDT Identified - We are currently investigating this issue.

Investigating major outage due to database failure.

Apr <var data-var='date'>10</var>, <var data-var='time'>14:36</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>10</var>, <var data-var='time'>13:19</var> PDT Monitoring - Tonkean App and Workflows should be back and running as expected. We're monitoring the system to make sure everything is operational. Apr <var data-var='date'>10</var>, <var data-var='time'>09:54</var> PDT Update - We are continuing to investigate this issue. Apr <var data-var='date'>10</var>, <var data-var='time'>09:54</var> PDT Investigating - We are investigating slowness experienced in the Tonkean app and in executing workflows.

Is Tonkean down?

Tonkean status is UP

Lastest published events

High errors rate in workflow execution + slowness in app loading

AI Agents are not responding

Scheduled maintenance

Tonkean Services, Components and Regions

Recent Tonkean events

High errors rate in workflow execution + slowness in app loading

AI Agents are not responding

Scheduled maintenance

Sharepoint data sources errors

Scheduled maintenance

Slowness in processing of Tonkean items, actions and interfaces