THIS IS A SCHEDULED EVENT Aug <var data-var='date'>24</var>, <var data-var='time'>14:00</var> - <var data-var='time'>22:00</var> UTC Jul <var data-var='date'>29</var>, <var data-var='time'>16:13</var> UTC Scheduled - The Sydney region will be completely unavailable during the following time window. This affects all SASE services specific to that region, including SGN Tunnel IPSEC endpoints and Static IP. Agents will connect to the next nearest region during this time.

SASE Scheduled Maintenance: Sydney

Aug <var data-var='date'>20</var>, <var data-var='time'>22:32</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>20</var>, <var data-var='time'>17:16</var> UTC Monitoring - A fix has been implemented and we are monitoring tunnels for any other issues. Please let us know if you are experiencing any further challenges. Aug <var data-var='date'>20</var>, <var data-var='time'>16:39</var> UTC Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>20</var>, <var data-var='time'>15:50</var> UTC Investigating - Investigating intermittent reported SASE tunnel errors. Please let technical support know if you are experiencing any tunnel issues. We will update here with more information shortly.

Intermittent SASE Tunnel Errors

Aug <var data-var='date'>18</var>, <var data-var='time'>05:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>18</var>, <var data-var='time'>03:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>12</var>, <var data-var='time'>18:28</var> UTC Scheduled - We will be performing scheduled maintenance on following Point of Presence (PoP). During this window, tunnels configured in these locations will be temporarily unavailable. - Dallas, USA - Washington D.C., USA - New York, USA - Atlanta, USA Customers with redundant configurations should not experience any disruption. Non-redundant tunnels terminating in these PoPs may experience downtime for part or all of the maintenance window.

SASE Scheduled Maintenance: Tunnels

Aug <var data-var='date'>16</var>, <var data-var='time'>15:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>16</var>, <var data-var='time'>07:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>13</var>, <var data-var='time'>19:36</var> UTC Scheduled - On Sunday August 16th, between the time window of 2:00 AM – 10:00 AM EDT, our facility provider has scheduled maintenance in Atlanta to upgrade hardware.  This is a routine hardware upgrade activity and should not impact customers with active/passive tunnels configured or static IPs configured across multiple PoPs. This maintenance is planned to be no more than 2.5 of downtime during the scheduled window and Todyl engineers will be on-call to monitor and support customers who experience any issues. If you do not have tunnel or static IP redundancies configured, reach out to your CSM or the Todyl support team for additional information or assistance.

Planned Hardware Upgrade Activity in Atlanta on August 16th

Aug <var data-var='date'>16</var>, <var data-var='time'>14:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>16</var>, <var data-var='time'>06:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>29</var>, <var data-var='time'>16:11</var> UTC Scheduled - The Atlanta region will be completely unavailable during the following time window. This affects all SASE services specific to that region, including SGN Tunnel IPSEC endpoints and Static IP. Agents will connect to the next nearest region during this time.

SASE Scheduled Maintenance: Atlanta

Aug <var data-var='date'>15</var>, <var data-var='time'>13:11</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>15</var>, <var data-var='time'>12:48</var> UTC Verifying - Verification is currently underway for the maintenance items. Aug <var data-var='date'>15</var>, <var data-var='time'>12:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>12</var>, <var data-var='time'>19:33</var> UTC Scheduled - We will be performing scheduled maintenance on our cloud environment. During this window, the management portal and the authentication service may be intermittently unreachable.

Todyl Cloud Environment Scheduled Maintenance

Aug <var data-var='date'>15</var>, <var data-var='time'>06:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>15</var>, <var data-var='time'>04:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>12</var>, <var data-var='time'>18:25</var> UTC Scheduled - We will be performing scheduled maintenance on following Point of Presence (PoP). During this window, tunnels configured in these locations will be temporarily unavailable. - Los Angeles, USA - Denver, USA - Chicago, USA Customers with redundant configurations should not experience any disruption. Non-redundant tunnels terminating in these PoPs may experience downtime for part or all of the maintenance window.

Aug <var data-var='date'> 8</var>, <var data-var='time'>16:14</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'> 8</var>, <var data-var='time'>15:59</var> UTC Verifying - Verification is currently underway for the maintenance items. Aug <var data-var='date'> 8</var>, <var data-var='time'>12:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'> 5</var>, <var data-var='time'>20:34</var> UTC Scheduled - We will be performing scheduled maintenance on our cloud environment. During this window, the management portal and the authentication service may be intermittently unreachable.

Aug <var data-var='date'> 7</var>, <var data-var='time'>05:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'> 7</var>, <var data-var='time'>04:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'> 5</var>, <var data-var='time'>20:29</var> UTC Scheduled - We will be performing scheduled maintenance on following Point of Presence (PoP). During this window, tunnels configured in these locations will be temporarily unavailable. - Queretaro, Mexico Customers with redundant configurations should not experience any disruption. Non-redundant tunnels terminating in these PoPs may experience downtime for part or all of the maintenance window.

Aug <var data-var='date'> 6</var>, <var data-var='time'>16:01</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'> 6</var>, <var data-var='time'>15:19</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'> 6</var>, <var data-var='time'>14:43</var> UTC Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'> 6</var>, <var data-var='time'>14:14</var> UTC Investigating - We are currently investigating reports of SASE connection errors on Windows Server 2012 devices. We will update as we have more information.

SASE connection failures for Windows Server 2012

Aug <var data-var='date'> 4</var>, <var data-var='time'>18:16</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'> 4</var>, <var data-var='time'>16:46</var> UTC Monitoring - The platform has returned to normal operation. We are continuing to monitor the situation to ensure full stability. Further updates will be provided if necessary. Aug <var data-var='date'> 4</var>, <var data-var='time'>16:25</var> UTC Identified - We have identified the root cause of the delays affecting SIEM results loading. The issue has been traced to a third-party upstream provider, and we are actively working with them to implement a fix. Users may continue to experience delays in SIEM results loading and cases appearing in the portal in the interim. Data integrity is not affected. Further updates will be provided as progress is made. Aug <var data-var='date'> 4</var>, <var data-var='time'>15:33</var> UTC Investigating - We are currently conducting an active investigation into an issue affecting our SIEM platform. Users may be experiencing delays when loading SIEM results. As a result of this delay, cases may also appear delayed in the portal. Our team is aware of the impact and is actively investigating the affected components. Further updates will be provided as more information becomes available.

SIEM Results Loading Delay Impacting Case Portal Visibility

Aug <var data-var='date'> 3</var>, <var data-var='time'>00:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'> 2</var>, <var data-var='time'>17:30</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>29</var>, <var data-var='time'>16:01</var> UTC Scheduled - The Hong Kong region will be completely unavailable during the following time window. This affects all SASE services specific to that region, including SGN Tunnel IPSEC endpoints and Static IP. Agents will connect to the next nearest region during this time.

SASE Scheduled Maintenance: Hong Kong

Jul <var data-var='date'>31</var>, <var data-var='time'>02:00</var> UTC Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>31</var>, <var data-var='time'>01:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>30</var>, <var data-var='time'>23:06</var> UTC Scheduled - We will be performing maintenance on SGN tonight from 9:00–10:00 PM ET (01:00–02:00 UTC). We do not anticipate any customer impact during this window. We will provide an update upon completion.

SGN Maintenance Window

Jul <var data-var='date'>30</var>, <var data-var='time'>22:54</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>30</var>, <var data-var='time'>21:48</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>30</var>, <var data-var='time'>18:04</var> UTC Identified - We have identified a subsequent issue related to the previous SASE incident. We are implementing a fix, but some users could still have intermittent SASE connection issues. We will update as soon as we know more on the fix.

SASE Reconnections - follow on

Jul <var data-var='date'>30</var>, <var data-var='time'>15:11</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>30</var>, <var data-var='time'>13:31</var> UTC Monitoring - The team has re-enabled tenant and policy changes across all PoPs and are monitoring the results. Jul <var data-var='date'>30</var>, <var data-var='time'>12:04</var> UTC Update - The team is now re-enabling tenant and policy changes across all PoPs. We will provide another update once the rollout is complete and stability has been validated. Jul <var data-var='date'>30</var>, <var data-var='time'>11:37</var> UTC Update - We have deployed a fix to stabilize the SASE environment and are continuing to investigate the underlying root cause. SASE connections are currently stable once established. As a precaution, tenant and policy updates remain temporarily paused while we complete validation and work toward full resolution. We will provide additional updates as new information becomes available. Jul <var data-var='date'>30</var>, <var data-var='time'>09:46</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'>30</var>, <var data-var='time'>09:21</var> UTC Investigating - We are currently investigating performance degradation affecting the Secure Global Network (SASE). A subset of users connecting may be experiencing disconnect and connect loop. Our engineering team is actively investigating the root cause.

SASE reconnections

Jul <var data-var='date'>27</var>, <var data-var='time'>21:33</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>27</var>, <var data-var='time'>18:25</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>27</var>, <var data-var='time'>15:54</var> UTC Update - The alert influx is impacting performance within the Todyl Management portal. Affected components include the portal experience and reporting widgets, which are experiencing slower than expected loading times. We are working to resolve this issue and will provide updates as the situation develops. Jul <var data-var='date'>27</var>, <var data-var='time'>15:34</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'>27</var>, <var data-var='time'>15:33</var> UTC Investigating - We are currently investigating an influx of Bash Shell Profile Modification alerts. These are all benign. Will update here as we know more.

Bash Shell Profile Modification alerts

Jul <var data-var='date'>24</var>, <var data-var='time'>20:46</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>24</var>, <var data-var='time'>14:01</var> UTC Monitoring - A third party provider has implemented a fix that appears to have resolved the intermittent URL failure. We will continue to monitor for any other issues and update here as needed. Jul <var data-var='date'>24</var>, <var data-var='time'>03:35</var> UTC Investigating - We are currently investigating reports of certain URLs intermittently failing to resolve correctly. Some domains, including those in the Google domain family, may be impacted. Our team is actively investigating and we will provide updates as more information becomes available.

Investigating intermittent URL resolution issue

Jul <var data-var='date'>23</var>, <var data-var='time'>20:19</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>23</var>, <var data-var='time'>18:38</var> UTC Monitoring - Microsoft is currently experiencing an incident affecting their M365 API. As a result, O365 data ingestion is delayed, temporarily impacting O365-sourced events. We are monitoring the upstream Microsoft incident and will update as it progresses.

Microsoft API Ingestion Delays

Jul <var data-var='date'>18</var>, <var data-var='time'>16:17</var> UTC Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>18</var>, <var data-var='time'>16:08</var> UTC Verifying - Verification is currently underway for the maintenance items. Jul <var data-var='date'>18</var>, <var data-var='time'>13:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>16</var>, <var data-var='time'>21:29</var> UTC Scheduled - We will be performing scheduled maintenance on our cloud environment. During this window, the management portal and the authentication service may be intermittently unreachable.

Jul <var data-var='date'> 7</var>, <var data-var='time'>22:34</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 7</var>, <var data-var='time'>20:31</var> UTC Monitoring - The connectivity issue with the Bogota Point of Presence (PoP) has been restored. We will continue to monitor to ensure no other issues. Jul <var data-var='date'> 7</var>, <var data-var='time'>19:51</var> UTC Identified - The connectivity issue was caused by a power outage from a third party provider. The provider is aware and currently working to restore power as soon as possible. Jul <var data-var='date'> 7</var>, <var data-var='time'>19:28</var> UTC Investigating - We are currently investigating connectivity issues with the Bogota Point of Presence (PoP). Devices will automatically route to the next nearest and fastest PoP and Static IP traffic will egress from the configured secondary location. Will update here as we the investigation continues.

Investigating Bogota Point of Presence (PoP) Connectivity Issue

Jul <var data-var='date'> 2</var>, <var data-var='time'>03:43</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 2</var>, <var data-var='time'>02:37</var> UTC Monitoring - It appears the 3rd party issues have been restored. Will continue monitoring until we are clear that it is fully resolved. Jul <var data-var='date'> 2</var>, <var data-var='time'>01:15</var> UTC Investigating - This issue is currently under investigation, but possibly related to a known Microsoft outage in the region - https://downdetector.com.au/status/microsoft-365/ We will post more information here as soon as we have better insight.

Investigating partial connectivity issues impacting Oceania region

Jun <var data-var='date'>24</var>, <var data-var='time'>00:43</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>23</var>, <var data-var='time'>23:21</var> UTC Monitoring - The connectivity issue with the Miami Point of Presence (PoP) has been restored. We will continue to monitor to ensure no other issues, but SASE, tunnels, and Static IP traffic should be connecting now. Jun <var data-var='date'>23</var>, <var data-var='time'>23:00</var> UTC Identified - The connectivity issue was caused by a power outage from a third party provider. The provider is aware and currently working to restore power as soon as possible. Jun <var data-var='date'>23</var>, <var data-var='time'>22:43</var> UTC Investigating - We are currently investigating connectivity issues with the Miami Point of Presence (PoP). Devices will automatically route to the next nearest and fastest PoP and Static IP traffic will egress from the configured secondary location. Will update here as we the investigation continues.

Investigating Miami Point of Presence (PoP) Connectivity Issue

Jun <var data-var='date'>22</var>, <var data-var='time'>21:01</var> UTC Resolved - This incident is considered resolved as we see no impact on Todyl portal anymore, despite Cloudflare still continuing to monitor the issue. Jun <var data-var='date'>22</var>, <var data-var='time'>17:03</var> UTC Update - Cloudflare has successfully mitigated the majority of congestion and packet drops. We are continuing to monitor any possible impacts to the Todyl portal. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/. Jun <var data-var='date'>22</var>, <var data-var='time'>16:48</var> UTC Monitoring - majority of congestion and packet drops has been mitigated. Services are stable, with only minor residual impact remaining. A small number of intermittent errors may still be observed for services with origins in North America. We are putting this into a monitoring state. Jun <var data-var='date'>22</var>, <var data-var='time'>16:48</var> UTC Identified - Cloudflare has successfully mitigated the majority of congestion and packet drops. We are continuing to monitor any possible impacts to the Todyl portal. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/. Jun <var data-var='date'>22</var>, <var data-var='time'>16:45</var> UTC Monitoring - Cloudflare has successfully mitigated the majority of congestion and packet drops. We are continuing to monitor any possible impacts to the Todyl portal. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/. Jun <var data-var='date'>22</var>, <var data-var='time'>14:56</var> UTC Identified - The issue has been identified and awaiting on a Cloudflare resolution. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/. Jun <var data-var='date'>22</var>, <var data-var='time'>14:52</var> UTC Investigating - Some partners may be experiencing intermittent latency when using the Todyl portal.  Currently there is a partial Cloudflare system error being reported, leading to the intermittent errors on the Todyl portal.  We are monitoring the situation closely and getting updates from Cloudflare.  We will be be providing regular updates. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/.

Cloudflare error causing intermittent portal delays

Jun <var data-var='date'>16</var>, <var data-var='time'>04:29</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>15</var>, <var data-var='time'>20:53</var> UTC Monitoring - Performance has returned to normal operation for the Secure Global Network (SASE) - Houston, HOU region. The system has self-recovered and affected users should no longer be experiencing elevated latency or reduced throughput. We are continuing to monitor the environment to ensure sustained stability. Jun <var data-var='date'>15</var>, <var data-var='time'>20:18</var> UTC Investigating - We are currently investigating performance degradation affecting the Secure Global Network (SASE) - Houston, HOU region. A subset of users connecting through this region may be experiencing elevated latency or reduced throughput. Our engineering team is actively investigating the root cause.

Secure Global Network (SASE) - Houston, HOU

Jun <var data-var='date'> 4</var>, <var data-var='time'>12:43</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'> 4</var>, <var data-var='time'>12:38</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 4</var>, <var data-var='time'>12:08</var> UTC Investigating - We are currently investigating an issue affecting Management Portal sign ins. A subset of users may experience intermittent failures when attempting to load the portal. All other Todyl services are operating normally. Our engineering team is actively investigating the root cause.

Is Todyl down?

Todyl status is UP

Lastest published events

Intermittent SASE Tunnel Errors

SASE Scheduled Maintenance: Tunnels

Planned Hardware Upgrade Activity in Atlanta on August 16th

Todyl Services, Components and Regions

Recent Todyl events

SASE Scheduled Maintenance: Sydney

Intermittent SASE Tunnel Errors

SASE Scheduled Maintenance: Tunnels

Planned Hardware Upgrade Activity in Atlanta on August 16th

SASE Scheduled Maintenance: Atlanta

Todyl Cloud Environment Scheduled Maintenance