Oct <var data-var='date'>29</var>, <var data-var='time'>15:00</var> MDT Completed - The scheduled maintenance has been completed. Oct <var data-var='date'>29</var>, <var data-var='time'>14:00</var> MDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>28</var>, <var data-var='time'>14:20</var> MDT Scheduled - On Wednesday, October 29th, we will be performing scheduled maintenance on the Protect production environment. The maintenance is expected to last about an hour. No downtime is expected, but some users may experience brief delays when accessing protect.threatx.io. This maintenance only affects customers who have been migrated to the Xplat environment. If you have not completed the migration, your service will not be impacted. We appreciate your understanding as we work to improve our services. If you have any questions or concerns, please contact our support team at support@threatx.com.

Scheduled Maintenance Notification

Aug <var data-var='date'>19</var>, <var data-var='time'>22:00</var> MDT Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>19</var>, <var data-var='time'>21:40</var> MDT Verifying - Verification is currently underway for the maintenance items. Aug <var data-var='date'>19</var>, <var data-var='time'>20:00</var> MDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>13</var>, <var data-var='time'>10:35</var> MDT Scheduled - On Tuesday, August 19th, we will be performing scheduled maintenance on the Protect production environment. The maintenance is expected to last about 2 hours. No downtime is expected, but some users may experience brief delays when accessing protect.threatx.io. This maintenance only affects customers who have been migrated to the Xplat environment. If you have not completed the migration, your service will not be impacted. We appreciate your understanding as we work to improve our services. If you have any questions or concerns, please contact our support team.

Aug <var data-var='date'>14</var>, <var data-var='time'>11:42</var> MDT Resolved - This incident has been resolved. Aug <var data-var='date'>14</var>, <var data-var='time'>10:41</var> MDT Monitoring - ThreatX Engineering has implemented a fix for the cause of the outage and we are closely monitoring for availability. Aug <var data-var='date'>14</var>, <var data-var='time'>10:23</var> MDT Identified - ThreatX Engineering has identified an issue causing an outage within the ThreatX Dashboard. We are actively working to resolve the cause of the outage.

Dashboard outage

Aug <var data-var='date'>12</var>, <var data-var='time'>20:22</var> MDT Completed - The scheduled maintenance has been completed. If you encounter any issues, please contact support. Aug <var data-var='date'>12</var>, <var data-var='time'>20:17</var> MDT Verifying - Verification is currently underway for the maintenance items. Aug <var data-var='date'>12</var>, <var data-var='time'>20:00</var> MDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'> 6</var>, <var data-var='time'>13:22</var> MDT Scheduled - On Tuesday, August 12th, we will be performing scheduled maintenance on the Protect production environment. The maintenance is expected to last about 1 hour. No downtime is expected, but some users may experience brief delays when accessing protect.threatx.io. This maintenance only affects customers who have been migrated to the Xplat environment. If you have not completed the migration, your service will not be impacted. We appreciate your understanding as we work to improve our services. If you have any questions or concerns, please contact our support team.

Jul <var data-var='date'>31</var>, <var data-var='time'>19:19</var> MDT Resolved - This issue has been identified and resolved. Jul <var data-var='date'>31</var>, <var data-var='time'>18:42</var> MDT Investigating - Ctrlx for protect.threatx.io is currently down and being investigated. Please note that this only impacts protect.threatx.io and not x.threatx.io. We will update as soon as we have additional information.

Protect Ctrlx Down

Jul <var data-var='date'>17</var>, <var data-var='time'>15:52</var> MDT Resolved - This incident is now resolved. Jul <var data-var='date'>17</var>, <var data-var='time'>15:48</var> MDT Update - The ThreatX Dashboard now appears to be available once again. ThreatX Engineering is actively monitoring the situation. Jul <var data-var='date'>17</var>, <var data-var='time'>15:28</var> MDT Update - ThreatX Engineering has identified another cause of the potential problems and is taking steps to address the root causes. The Dashboard appears to be intermittently unavailable during this incident. Jul <var data-var='date'>17</var>, <var data-var='time'>15:13</var> MDT Monitoring - ThreatX Engineering has found and resolved a potential source of the problem. ThreatX Engineering is continuing to monitor the situation closely. Jul <var data-var='date'>17</var>, <var data-var='time'>15:06</var> MDT Investigating - The ThreatX Dashboard appears to be unavailable. ThreatX Engineering is actively investigating. At this point in time we have no reason to believe protection of customer sites is affected.

ThreatX Dashboard unavailable

Mar <var data-var='date'>11</var>, <var data-var='time'>12:00</var> MDT Resolved - The incident has been resolved and all service are operating normally. Mar <var data-var='date'>11</var>, <var data-var='time'>11:06</var> MDT Investigating - We are currently investigating an issue with API that is also impacting access to Ctrlx. Sensors remain unaffected.

API and Ctrlx Outage

Feb <var data-var='date'>27</var>, <var data-var='time'>16:31</var> MST Resolved - This incident is now resolved. Feb <var data-var='date'>27</var>, <var data-var='time'>16:00</var> MST Update - ThreatX Engineering is continuing to investigate this issue. At this time we believe the problem is limited to the ThreatX Dashbaord. Hosted WAF operations are not affected. ThreatX Engineering is actively investigating this issue and will post an update as more information becomes available. Feb <var data-var='date'>27</var>, <var data-var='time'>15:27</var> MST Investigating - ThreatX Engineering is investigating a backend service outage. Most backend services appear to be affected. Hosted WAF operations are not affected. ThreatX Engineering is actively investigating this issue and will post an update as more information becomes available.

Backend service outage

Feb <var data-var='date'>25</var>, <var data-var='time'>12:17</var> MST Resolved - ThreatX Engineering believes the problem has been addressed. All customers should now be able to login without delays through all SSO providers. Feb <var data-var='date'>25</var>, <var data-var='time'>10:29</var> MST Update - ThreatX Engineering has observed the connection issues appear to be subsiding. At this time we are no longer observing client timeouts for any customer attempting to log in to the ThreatX Dashboard. We will continue to monitor and troubleshoot for slow login symptoms. Feb <var data-var='date'>25</var>, <var data-var='time'>09:57</var> MST Update - ThreatX Engineering is continuing to investigate this issue. We believe we have narrowed down the problem to customers using Microsoft Azure SSO to log in to the ThreatX Dashboard. All other functionality including application protection appears to be unaffected at this time. ThreatX Engineering is actively investigating this issue and will provide updates as more information becomes available. Feb <var data-var='date'>25</var>, <var data-var='time'>09:46</var> MST Investigating - ThreatX Engineering has identified degraded performance and intermittent timeouts affecting the ThreatX Dashboard page. The problem appears to be an intermittent issue at this time. ThreatX Engineering is actively investigating this issue and will provide updates as more information becomes available.

Degraded performance on ThreatX Dashboard

Feb <var data-var='date'> 5</var>, <var data-var='time'>10:33</var> MST Resolved - This incident has been resolved. Feb <var data-var='date'> 5</var>, <var data-var='time'>10:32</var> MST Update - All ThreatX services are operating within normal parameters. This incident is now resolved. Feb <var data-var='date'> 5</var>, <var data-var='time'>10:20</var> MST Update - ThreatX Engineering is continuing to monitor closely as our hosting provider continues to restore connectivity to affected services. Some processing delays may continue to exist as affected systems recover. We will update the status page once all services are operating within normal parameters. Feb <var data-var='date'> 5</var>, <var data-var='time'>10:05</var> MST Monitoring - Our hosting provider, IBM, has begun restoring services that were impacting ThreatX. As systems come back online, some processing delays may occur. ThreatX Engineering is monitoring closely and will update the status page once all services are operating within normal parameters. Feb <var data-var='date'> 5</var>, <var data-var='time'>08:50</var> MST Investigating - We are currently experiencing an outage impacting our main platform and API services due to an issue with our hosting provider, IBM. Our team is actively working with IBM to resolve the problem and restore service as quickly as possible. All sensors and protection remain fully operational. We will provide updates as soon as we have more information.

Service Outage

Jan <var data-var='date'>23</var>, <var data-var='time'>16:15</var> MST Resolved - Issue resolved, all looks green across the board. Jan <var data-var='date'>23</var>, <var data-var='time'>14:57</var> MST Monitoring - Services are operating normally and we are continuing to monitor. Jan <var data-var='date'>23</var>, <var data-var='time'>13:50</var> MST Investigating - ThreatX Engineering has detected intermittent degraded performance affecting ThreatX hosted services (Dashboard, API, Gateway, and Risk Analytics Engine). We are currently investigating the issue. At this time we have no reason to believe protection of customer sites (ie Hosted WAF instances) is affected in any way.

Intermittent Service Outages / Delays

Oct <var data-var='date'>22</var>, <var data-var='time'>09:16</var> MDT Resolved - Incident has been resolved, all services are operating normally. Sep <var data-var='date'>26</var>, <var data-var='time'>18:18</var> MDT Monitoring - ThreatX Engineering has identified and removed a faulty service instance. Performance now appears to be within nominal levels. ThreatX Engineering is monitoring the system for any potential additional delays. Sep <var data-var='date'>26</var>, <var data-var='time'>08:33</var> MDT Investigating - ThreatX Engineering is actively investigating an issue in which sensor updates are delayed in taking effect on customer sensors. At this time we believe updates do eventually reach customer sensors after an atypical delay of up to several minutes. This issue appears to be intermittent for affected customers. Existing security protections and site configurations already in place are not affected by this issue. ThreatX Engineering will provide updates as we learn more.

Processing delays for sensor updates

Aug <var data-var='date'> 9</var>, <var data-var='time'>22:54</var> MDT Resolved - ThreatX Operations has confirmed this incident is no longer active. Aug <var data-var='date'> 9</var>, <var data-var='time'>21:37</var> MDT Identified - ThreatX Operations has identified the issue as originating in a cloud services provider and affecting a small subset of ThreatX customers. ThreatX SOC is contacting affected customers directly to coordinate responses. Aug <var data-var='date'> 9</var>, <var data-var='time'>21:23</var> MDT Investigating - ThreatX Security Operations has detected what appears to be a performance degradation affecting hosted sensors in certain regions in Canada. Affected customers may experience intermittent degraded performance. ThreatX Operations is actively investigating this incident to determine the extent of the impact and what corrective action can be taken.

Scaler performance degradation in Central Canada region

Jul <var data-var='date'>21</var>, <var data-var='time'>15:13</var> MDT Resolved - This incident has been resolved. Jul <var data-var='date'>21</var>, <var data-var='time'>13:39</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>21</var>, <var data-var='time'>11:35</var> MDT Update - We are continuing to investigate this issue. Jul <var data-var='date'>21</var>, <var data-var='time'>08:50</var> MDT Investigating - We are currently experiencing issues with a 3rd party vendor that is resulting in delays and intermittent outages to threatx services. We are actively troubleshooting and have reached out to our vendor for support. Please note that active sensors are still functioning/protecting, however new sensors will fail to initialize. We will update this incident as more information becomes available.

Intermittent Service Outages/Delays

Jun <var data-var='date'> 4</var>, <var data-var='time'>13:22</var> MDT Resolved - This incident has been resolved. Jun <var data-var='date'> 4</var>, <var data-var='time'>10:28</var> MDT Investigating - ThreatX Engineering is currently investigating a partial outage in customer Analytics Events data. We have confirmed some analytics events data for some customers are either delayed or experiencing "gaps" in reporting. We do not yet have confirmation of which customers are affected and for which event types and time periods. We will update this status page with more information as it becomes available. Update: To clarify, ThreatX protection remains in effect for all customers. This incident applies only to reported analytics event data.

Partial Analytics Events outage

Dec <var data-var='date'> 7</var>, <var data-var='time'>00:20</var> MST Completed - The scheduled maintenance has been completed. Dec <var data-var='date'> 7</var>, <var data-var='time'>00:15</var> MST Verifying - The upgrade process is complete. We are in the process of verifying and hope to have the maintenance resolved within the next 30 minutes. Dec <var data-var='date'> 7</var>, <var data-var='time'>00:00</var> MST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>30</var>, <var data-var='time'>15:00</var> MST Scheduled - On Thursday December 7th we will be performing database maintenance to add disk space. We have established a 02:00 to 05:00 EST window with our vendor to perform this upgrade. No sensors or platform components should be affected by this change. We will update the status page upon start and completion. If you have any questions or concerns, please reach out to support@threatx.com.

Database Maintenance

Dec <var data-var='date'> 6</var>, <var data-var='time'>00:30</var> MST Completed - The scheduled maintenance has been completed. Dec <var data-var='date'> 6</var>, <var data-var='time'>00:21</var> MST Verifying - The upgrade process is complete. We are in the process of verifying and hope to have the maintenance resolved within the next 30 minutes. Dec <var data-var='date'> 6</var>, <var data-var='time'>00:02</var> MST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>30</var>, <var data-var='time'>14:59</var> MST Scheduled - On Wednesday December 6th we will be performing database maintenance to add disk space. We have established a 02:00 to 05:00 EST window with our vendor to perform this upgrade. No sensors or platform components should be affected by this change. We will update the status page upon start and completion. If you have any questions or concerns, please reach out to support@threatx.com.

Dec <var data-var='date'> 5</var>, <var data-var='time'>00:53</var> MST Completed - Maintenance complete. Dec <var data-var='date'> 5</var>, <var data-var='time'>00:23</var> MST Verifying - The upgrade process is complete. We are in the process of verifying and hope to have the maintenance resolved within the next 30 minutes. Dec <var data-var='date'> 5</var>, <var data-var='time'>00:00</var> MST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>30</var>, <var data-var='time'>14:58</var> MST Scheduled - On Tuesday December 5th we will be performing database maintenance to add disk space. We have established a 02:00 to 05:00 EST window with our vendor to perform this upgrade. No sensors or platform components should be affected by this change. We will update the status page upon start and completion. If you have any questions or concerns, please reach out to support@threatx.com.

Nov <var data-var='date'>10</var>, <var data-var='time'>17:32</var> MST Resolved - After an additional period of monitoring, ThreatX Engineering is satisfied that all affected customers have now resumed normal risk processing operations. This incident is now resolved. Nov <var data-var='date'>10</var>, <var data-var='time'>17:22</var> MST Monitoring - ThreatX Engineering has identified the likely source of the problem and has implemented a fix. Processing rates appear to be returning to normal. ThreatX Engineering is monitoring closely. Nov <var data-var='date'>10</var>, <var data-var='time'>16:37</var> MST Investigating - ThreatX Engineering is investigating an issue in which a subset of customer traffic is not being properly processed for risk-based blocking. This outage affects a subset of customers and does not affect other forms of blocking such as request-based blocking or access-list-based blocking. ThreatX Engineering is actively investigating and will post an update as soon as more information is known.

Partial Risk-based blocking outage

Oct <var data-var='date'>24</var>, <var data-var='time'>11:59</var> MDT Resolved - This incident has been resolved. Oct <var data-var='date'>24</var>, <var data-var='time'>11:28</var> MDT Update - We are continuing to monitor for any further issues. Oct <var data-var='date'>24</var>, <var data-var='time'>11:25</var> MDT Monitoring - A fix has been implemented and ThreatX Engineering is monitoring. Oct <var data-var='date'>24</var>, <var data-var='time'>11:20</var> MDT Identified - ThreatX Engineering has identified a possible cause of the outage and is working on restoring the security rules. Oct <var data-var='date'>24</var>, <var data-var='time'>10:50</var> MDT Investigating - A subset of security rules are not being enforced by ThreatX sensors due to an unknown failure in the security rule distribution mechanism. ThreatX Engineering is actively investigating.

Partial security rule outage

Oct <var data-var='date'>23</var>, <var data-var='time'>16:01</var> MDT Resolved - ThreatX Engineering has confirmed the mitigating change was executed successfully and normal access to the ThreatX Dashboard and Provisioning API has been restored. Oct <var data-var='date'>23</var>, <var data-var='time'>15:49</var> MDT Monitoring - ThreatX Engineering has implemented a change to address the outage and is monitoring system availability. Oct <var data-var='date'>23</var>, <var data-var='time'>15:43</var> MDT Identified - ThreatX Engineering has identified a likely cause of the outage and is currently working on a solution. Oct <var data-var='date'>23</var>, <var data-var='time'>15:33</var> MDT Update - We are continuing to investigate this issue. Oct <var data-var='date'>23</var>, <var data-var='time'>15:33</var> MDT Investigating - Users of the ThreatX Dashboard have reported being unable to log in. ThreatX Engineering is actively investigating this issue.

ThreatX Dashboard outage

Sep <var data-var='date'>25</var>, <var data-var='time'>07:44</var> MDT Resolved - This incident has been resolved. Sep <var data-var='date'>25</var>, <var data-var='time'>07:17</var> MDT Investigating - We are aware of an issue with domains resolving to txproxy.com. We are currently investigating and hope to have the issue resolved shortly. This impacts any sensor traffic flow routed through txproxy.com.

Issues With txproxy.com

Jul <var data-var='date'>12</var>, <var data-var='time'>19:01</var> MDT Resolved - INAP has now fully restored their NYJ data center. All services are operating normally. We will continue to monitor and re-issue a status alert if new issues arise. Jul <var data-var='date'>11</var>, <var data-var='time'>19:35</var> MDT Update - Latest update from INAP: Update - The Secaucus data center remains powered down at this time, per the fire marshal. Evocative onsite management met with the fire marshal and electrical inspectors to review the current state of the facility. The fire marshal has reviewed the current level of progress on the cleaning and requested additional remediation actions around the fire system components. The follow up inspection from the fire marshal is 9:00 AM EDT, Wednesday. The Evocative team is bringing on additional personnel and vendors to comply completely with the additional requests by this deadline. In preparation for being able to allow clients onsite, the fire marshal has stated that Evocative must perform a full test of the fire/life safety systems. This must be performed after utility power has been restored and fire system components replaced. Vendors have already been scheduled and are standing by to complete this work tomorrow. If there are no significant changes in the timeline, the earliest INAP will be allowed back into the site to power up servers will be later in the day on Wednesday. We anticipate the next update of significance will be provided by Evocative Wednesday morning, and will keep you updated as information becomes available. Jul 11, 2023 - 16:04 CDT Jul <var data-var='date'>11</var>, <var data-var='time'>11:32</var> MDT Update - Latest update from INAP: Update - Dear Valued Customer, Evocative has worked with their remediation vendor through the night to clean the UPS' at the request of the fire marshal. They have made significant progress and aim to have the cleaning completed by mid-day. There is currently a 2PM EDT meeting scheduled, during which Evocative staff and local township officials will perform a walk-through of the site. Evocative will need approvals from the Mayor, the fire marshal, and the local utility before the site can be re-energized. Depending on their inspections and approvals, Evocative staff will then be able to access the site and initiate the sequence to bring all data center systems back online. Evocative will be providing us a status update after this meeting takes place. While we do not have an ETA for when power will be fully restored, the re-energizing process can take 4-5 hours after it is initiated. Regards, INAP Cloud Support Jul 11, 2023 - 11:35 CDT Jul <var data-var='date'>11</var>, <var data-var='time'>10:20</var> MDT Update - Latest update from INAP: Update - Dear Valued Customer, As we reported earlier, the incident is related to a UPS system in the Secaucus data center owned and operated by Evocative. There was a fire in the UPS room, which was extinguished, but the incident resulted in a shutdown of all power to the facility and the involvement of the local fire department. This is the reason that redundant power systems did not keep any of the servers up and operational. The fire marshal is requiring extensive cleaning of the affected systems and areas before the site can be re-energized. That work is in progress. We were informed that Evocative will need to get an OK from the Mayor, the fire marshal, and the local utility to re-energize the site; depending on their inspection and approval, they can enter the site and begin to sequence bringing all data center systems back online. There is currently no ETA, but we will continue to provide updates. Once that is done, we have our teams standing by both locally and within our engineering and support organizations to ensure that all servers and services are brought back online as swiftly as possible. Regards, INAP Cloud Support Jul 11, 2023 - 08:06 CDT Jul <var data-var='date'>10</var>, <var data-var='time'>21:54</var> MDT Update - Further updates from INAP below. It is looking likely that there will be no resolution before 08:00 EDT tomorrow. Update - Our provider has shared further details about the incident and current status. A small electrical failure occurred with one of the redundant UPS devices which began smoking and led to a small fire in the UPS room. The fire department was dispatched and the fire extinguished quickly. The fire department subsequently cut power to the datacenter and disabled the generators while they and the utility company inspected the electrical system. The fire marshal is requiring extensive cleaning of the UPS devices and rooms before the site can be reenergized. There is a vendor onsite currently who will be performing the cleanup. The next update from our provider will be at 8:00 AM EDT unless there are significant changes overnight. We will continue to pass along information as we receive it. Jul 10, 2023 - 22:21 CDT Jul <var data-var='date'>10</var>, <var data-var='time'>18:39</var> MDT Update - See below for an additional update from INAP. Update - Dear Valued Customer, We have spoken with our provider and received additional details and a tentative ETA. There was a problem within the UPS system that required the site to be fully powered down. Vendors are onsite and working to resolve the issue. The current estimate is that power should be restored within the next several hours. We will continue to work with our provider and post regular updates. Regards, INAP Cloud Support Jul 10, 2023 - 18:59 CDT Jul <var data-var='date'>10</var>, <var data-var='time'>17:06</var> MDT Update - From INAP Update - Dear Valued Customer, We are seeing intermittent power availability issues within this datacenter and are working diligently with the provider on a resolution. Regards, INAP Cloud Support Jul 10, 2023 - 18:02 CDT Jul <var data-var='date'>10</var>, <var data-var='time'>17:02</var> MDT Investigating - INAP is experiencing network connectivity issues in their NYJ data centers which may be impacting ThreatX hosted sensors in that region. Please note that this is only impacting INAP sensors in NYJ, AWS sensors are not impacted. We will update this page as more information becomes available.

INAP Outage

Jun <var data-var='date'> 8</var>, <var data-var='time'>00:25</var> MDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'> 8</var>, <var data-var='time'>00:00</var> MDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>30</var>, <var data-var='time'>15:22</var> MDT Scheduled - On Thursday, June 8th between the hours 12am and 3am MDT, we will be performing required database maintenance on a single node in our cluster. We do not anticipate any down time during this period however, you may see slight delays or increased load times in core services like CtrlX. If you have any questions or concerns, please contact support@threatx.com

Database Maintenance 06-08-2023

Jun <var data-var='date'> 7</var>, <var data-var='time'>00:34</var> MDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'> 7</var>, <var data-var='time'>00:00</var> MDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>30</var>, <var data-var='time'>15:21</var> MDT Scheduled - On Wednesday, June 7th between the hours 12am and 3am MDT, we will be performing required database maintenance on a single node in our cluster. We do not anticipate any down time during this period however, you may see slight delays or increased load times in core services like CtrlX. If you have any questions or concerns, please contact support@threatx.com

Is ThreatX down?

ThreatX status is UP

Lastest published events

Scheduled Maintenance Notification

Scheduled Maintenance Notification

Dashboard outage

ThreatX Services, Components and Regions

Recent ThreatX events

Scheduled Maintenance Notification

Scheduled Maintenance Notification

Dashboard outage

Scheduled Maintenance Notification

Protect Ctrlx Down

ThreatX Dashboard unavailable