May <var data-var='date'>23</var>, <var data-var='time'>00:29</var> CDT Resolved - We've identified that traffic has rerouted successfully, and all packet loss has cleared. May <var data-var='date'>23</var>, <var data-var='time'>00:21</var> CDT Monitoring - Summit engineers have taken action to reroute traffic from the problematic carrier, and we are observing the packet loss clearing. Local traffic did not appear to be impacted. We are monitoring for continued stability. May <var data-var='date'>23</var>, <var data-var='time'>00:10</var> CDT Identified - Summit engineers believe this was isolated to transpacific traffic via a single carrier. We have taken actions to reroute traffic. May <var data-var='date'>22</var>, <var data-var='time'>23:28</var> CDT Investigating - The Summit engineering team is investigating Network packet loss and latency in the Sydney region.

Network Instablity SYD

Mar <var data-var='date'>31</var>, <var data-var='time'>22:05</var> CDT Resolved - Waveserver pair between ORD1 and ORD6 was restored to service after a cold boot of the appliances. Traffic is now being passed properly once again. Mar <var data-var='date'>31</var>, <var data-var='time'>21:51</var> CDT Identified - Limited degraded network performance for select customers with wave circuit transit between Elk Grove Village (DF-CH1 - Digital Realty Trust CH1), and Chicago 350 Cermak (EQ-CH1).

Internal Network Transit Event

Nov <var data-var='date'>25</var>, <var data-var='time'>18:34</var> CST Resolved - Microsoft 365 services continue to return to functional status and our email and voice communications are back online. We will monitor these issues throughout the evening in the event of any lingering issues. Thank you for your patience. Nov <var data-var='date'>25</var>, <var data-var='time'>12:19</var> CST Monitoring - We are currently experiencing an outage affecting Microsoft 365 services, including email and voice communications. This disruption may impact our ability to communicate effectively, including ticketing responses. Microsoft has acknowledged the issue and is actively working to resolve it. We will provide updates as more information becomes available.

Microsft 365 Outage affecting Summit communications.

Sep <var data-var='date'>18</var>, <var data-var='time'>08:22</var> CDT Resolved - Deft Engineers have identified an issue with one of our transit providers and are working on shifting traffic off the carrier. They will conduct further investigation with the carrier and provide updates as more information becomes available.

Network Carrier Incident

Jun <var data-var='date'>24</var>, <var data-var='time'>10:50</var> CDT Resolved - This incident has been resolved. Jun <var data-var='date'>20</var>, <var data-var='time'>16:23</var> CDT Monitoring - We have applied a vendor-suggested fix that appears to have mitigated the issues. We will continue to monitor and test to validate resolution. Jun <var data-var='date'>20</var>, <var data-var='time'>12:43</var> CDT Identified - Engineers are implementing a workaround and will continue to test resolution of the issue. Jun <var data-var='date'>20</var>, <var data-var='time'>11:44</var> CDT Update - We have isolated the issue to our Chicago Managed Object Storage environment Jun <var data-var='date'>20</var>, <var data-var='time'>10:33</var> CDT Investigating - Engineers are currently investigating an issue with our Managed Object Storage.

Managed Object Storage Degredation

Apr <var data-var='date'>19</var>, <var data-var='time'>19:27</var> CDT Resolved - This incident has been resolved. Apr <var data-var='date'>19</var>, <var data-var='time'>19:27</var> CDT Update - The incident has been resolved. Please contact Deft if you have any issues. Apr <var data-var='date'>19</var>, <var data-var='time'>16:20</var> CDT Update - We are continuing to work on a fix for this issue. Apr <var data-var='date'>19</var>, <var data-var='time'>11:00</var> CDT Identified - The issue has been identified and a fix is being implemented.

IAD Network Incident

Aug <var data-var='date'>11</var>, <var data-var='time'>19:34</var> CDT Resolved - This incident has been resolved. Aug <var data-var='date'>11</var>, <var data-var='time'>16:45</var> CDT Update - Service continues to be stable at this time. Deft Network Engineering will re-enable the circuit in question after business hours and continue to monitor. Aug <var data-var='date'>11</var>, <var data-var='time'>13:23</var> CDT Monitoring - Deft's 24/7 NOC has monitored an event with one of our upstream IP Transit circuits. Customers may have noticed a brief interruption in a portion of their internet traffic. We have disabled this circuit for the time being to mitigate any further impact.

ORD IP Transit

Apr <var data-var='date'>11</var>, <var data-var='time'>13:32</var> CDT Resolved - This incident has been resolved. Apr <var data-var='date'>11</var>, <var data-var='time'>13:32</var> CDT Update - We are continuing to monitor for any further issues. Apr <var data-var='date'> 1</var>, <var data-var='time'>11:55</var> CDT Update - We are continuing to monitor for any further issues. Apr <var data-var='date'> 1</var>, <var data-var='time'>11:53</var> CDT Monitoring - Currently our Looking glass, https://lg.deft.com is down for maintenance

Looking Glass - Undergoing Maintenance

Aug <var data-var='date'>30</var>, <var data-var='time'>10:37</var> CDT Resolved - CenturyLink has advised that they are taking corrective action and expect things to be clearing soon if not already. For our part, we are leaving them in a non-preferred state for the time being. Aug <var data-var='date'>30</var>, <var data-var='time'>09:52</var> CDT Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>30</var>, <var data-var='time'>09:50</var> CDT Update - CenturyLink have advised they have identified the issue and are taking corrective measures. Aug <var data-var='date'>30</var>, <var data-var='time'>08:16</var> CDT Identified - SCTG NOC has monitored trouble with the CenturyLink / Level 3 (AS3356) network. We have taken steps to mitigate but there are still issues remaining due to the size and scope of their network.

CenturyLink / Level3 Routing Incident

Aug <var data-var='date'> 4</var>, <var data-var='time'>13:32</var> CDT Resolved - Everything continues to be stable. We will follow up with vendor but expect no further issues. Aug <var data-var='date'> 4</var>, <var data-var='time'>09:59</var> CDT Monitoring - SCTG Network Operations have identified a brief interruption to one of our redundant dark fiber paths between 9:29 AM CDT and 9:42 AM CDT. All services are restored. We are monitoring and pursuing this with the vendor.

Chicago area dark fiber brief interruption

Jan <var data-var='date'>29</var>, <var data-var='time'>18:03</var> CST Resolved - We've moved back to our primary provider. Jan <var data-var='date'>29</var>, <var data-var='time'>17:01</var> CST Identified - Calls to our main IVR and NOC have been rerouted to an alternate phone provider while we troubleshoot the issue with our primary vendor.

Inbound Phone Issues

Dec <var data-var='date'>20</var>, <var data-var='time'>10:55</var> CST Resolved - Storage latency has returned to normal levels. Enterprise Cloud is operating normally at this time. Dec <var data-var='date'>20</var>, <var data-var='time'>10:43</var> CST Update - We are continuing to investigate this issue. Dec <var data-var='date'>20</var>, <var data-var='time'>10:43</var> CST Investigating - ServerCentral Managed Services staff are investigating reports of increased storage latency affected Enterprise Cloud ORD.

Increased Latency

Dec <var data-var='date'>14</var>, <var data-var='time'>22:00</var> CST Resolved - This issue was narrowed down to an incorrect configuration applied earlier this evening and has been resolved. Dec <var data-var='date'>14</var>, <var data-var='time'>21:39</var> CST Investigating - ServerCentral Turing Group is exploring a higher than usual amount of bandwidth entering our network. Some customers may experience slower than normal network response times. We are currently investigating this issue.

Packet Loss on Enterprise Cloud ORD6

Sep <var data-var='date'>20</var>, <var data-var='time'>00:01</var> CDT Resolved - This incident has been resolved. Sep <var data-var='date'>19</var>, <var data-var='time'>23:48</var> CDT Identified - We have identified the issue and are working to restore service. Sep <var data-var='date'>19</var>, <var data-var='time'>23:42</var> CDT Investigating - We are investigating reports of an issue relating to connecting to the Veeam Cloud Connect repository in our Elk Grove Village area data center.

Veeam Cloud Connect - ORD6

Sep <var data-var='date'>13</var>, <var data-var='time'>07:34</var> CDT Resolved - This incident has been resolved. Sep <var data-var='date'>13</var>, <var data-var='time'>01:19</var> CDT Monitoring - Services have been restored and are resuming normal activity. We are continuing to monitor the platform to ensure stability. Sep <var data-var='date'>13</var>, <var data-var='time'>01:13</var> CDT Identified - We have identified the issue and are working to restore service. Sep <var data-var='date'>13</var>, <var data-var='time'>00:47</var> CDT Investigating - We are investigating reports of an issue relating to connecting to the Veeam Cloud Connect repository in our Elk Grove Village area data center.

Sep <var data-var='date'> 6</var>, <var data-var='time'>12:22</var> CDT Resolved - This issue has been deemed resolved by our Managed Services team and all functionality is operating normally. Sep <var data-var='date'> 6</var>, <var data-var='time'>11:40</var> CDT Update - Services have been restored and are resuming normal activity. We are continuing to monitor the platform to ensure stability. Sep <var data-var='date'> 6</var>, <var data-var='time'>11:34</var> CDT Identified - We have identified the issue and are working to restore service. Sep <var data-var='date'> 6</var>, <var data-var='time'>11:28</var> CDT Investigating - We are investigating reports of an issue relating to issues connecting to the Veeam Cloud Connect repository in our Elk Grove Village area datacenter.

Aug <var data-var='date'>21</var>, <var data-var='time'>22:17</var> CDT Resolved - This issue has been resolved. All services are now operational. Aug <var data-var='date'>21</var>, <var data-var='time'>22:03</var> CDT Monitoring - We have resolved the issue and all system functionality is being restored. Aug <var data-var='date'>21</var>, <var data-var='time'>21:53</var> CDT Identified - We have Identified the issue and are actively working to resolve it. Aug <var data-var='date'>21</var>, <var data-var='time'>21:45</var> CDT Investigating - ServerCentral Staff are currently examining an issue related to Veeam Cloud Connect and Backup Repositories located in our Elk Grove Village Datacenter. Further details will be posted to status.servercentral.com.

VCC-ORD6

Jun <var data-var='date'>29</var>, <var data-var='time'>13:30</var> CDT Resolved - Comcast has implemented a temporary solution as they deal with multiple fiber cuts on their backbone Jun <var data-var='date'>29</var>, <var data-var='time'>12:11</var> CDT Investigating - The ServerCentral NOC monitoring systems have detected issues within Comcast's network. We are working to minimize the impact to our customers.

Comcast Backbone

Mar <var data-var='date'> 2</var>, <var data-var='time'>08:14</var> CST Resolved - Repairs are complete on our DWDM system, and we are back to normal. Mar <var data-var='date'> 2</var>, <var data-var='time'>06:08</var> CST Identified - ServerCentral has disabled a segment of our DWDM system on riser fiber between Floors 1 and 5 at 350 E Cermak. No customer capacity has been affected, but we have lost some redundancy within the building. We expect to have it repaired shortly.

350 Cermak Fiber Riser, Loss of Redundancy

Feb <var data-var='date'>22</var>, <var data-var='time'>09:39</var> CST Resolved - Efforts continue in working with our vendor to determine the root cause of the incident. Feb <var data-var='date'>21</var>, <var data-var='time'>18:22</var> CST Update - Senior Network Engineering continues to monitor the situation closely. Feb <var data-var='date'>21</var>, <var data-var='time'>18:20</var> CST Monitoring - We are placing the affected devices under a configuration freeze until such time as our consultation with our vendor reaches a positive conclusion. Feb <var data-var='date'>21</var>, <var data-var='time'>18:08</var> CST Investigating - We saw the routing process on one of our two Core Routers in 350 Cermak bounce at 23:56 UTC today. We are investigating the event with our vendor.

Chicago 350 Cermak Core Router Process Bounced

Feb <var data-var='date'>16</var>, <var data-var='time'>11:05</var> CST Resolved - We are continuing our follow up investigation with our vendor, but expect no further impact at this time. Feb <var data-var='date'>16</var>, <var data-var='time'>08:00</var> CST Monitoring - We saw the routing process on one of our two Core Routers in 350 Cermak bounce at 13:25 UTC today. We are investigating the event with our vendor.

Nov <var data-var='date'> 6</var>, <var data-var='time'>13:24</var> CST Resolved - Level3 has stopped leaking Comcast routes to NTT. Nov <var data-var='date'> 6</var>, <var data-var='time'>13:02</var> CST Monitoring - ServerCentral is now blocking Comcast routes that were leaked from Level3 to NTT. Level3 is still leaking these routes, which may be causing congestion on Level3's network. Nov <var data-var='date'> 6</var>, <var data-var='time'>12:53</var> CST Update - ServerCentral has removed the affected routes from this issue, and traffic should begin flowing normally to Comcast at this time. Nov <var data-var='date'> 6</var>, <var data-var='time'>12:26</var> CST Identified - ServerCentral has identified an issue with Comcast, and is working with them to resolve. We are working to reroute the affected traffic. Nov <var data-var='date'> 6</var>, <var data-var='time'>12:19</var> CST Investigating - We are investigating reports of latency on our Network

Network Latency

Sep <var data-var='date'> 5</var>, <var data-var='time'>18:08</var> CDT Resolved - The incident has been resolved at approximately 6:05PM CDT. We apologize for the inconvenience this may have caused. Sep <var data-var='date'> 5</var>, <var data-var='time'>17:59</var> CDT Identified - Our engineers have identified the issue and are working to bring the environment back online. Sep <var data-var='date'> 5</var>, <var data-var='time'>17:42</var> CDT Update - Our engineers are still currently investigating the interruption of service. Sep <var data-var='date'> 5</var>, <var data-var='time'>17:07</var> CDT Investigating - At approximately 4:50PM Central time, the Enterprise Cloud ORD6 availability zone experienced a partial outage. Engineers are investigating the cause now.

Enterprise Cloud ORD6

Aug <var data-var='date'>24</var>, <var data-var='time'>15:20</var> CDT Resolved - The Network event is resolved. One router was affected resulting in an outage for single homed customers. We apologize for the inconvenience this may have caused. Aug <var data-var='date'>24</var>, <var data-var='time'>14:57</var> CDT Investigating - We are investigating a network event in Tokyo, Japan. Senior network engineering staff are currently working to isolate and resolve the issue.

CR2.TK01

Aug <var data-var='date'>21</var>, <var data-var='time'>14:27</var> CDT Resolved - The Network Software event is resolved. One router was affected resulting in an outage for single homed customers. We apologize for the inconvenience this may have caused. Aug <var data-var='date'>21</var>, <var data-var='time'>13:55</var> CDT Investigating - We are currently investigating this issue.

We are investigating Network Software issue in SJC1 San Jose, CA  affecting customers beginning at 1:53 PM CDT. Technicians are currently working to determine the problem. An update will be provided in 30 minutes.

Is Deft down?

Deft status is UP

Lastest published events

Network Instablity SYD

Internal Network Transit Event

Microsft 365 Outage affecting Summit communications.

Deft Services, Components and Regions

Recent Deft events

Network Instablity SYD

Internal Network Transit Event

Microsft 365 Outage affecting Summit communications.

Network Carrier Incident

Managed Object Storage Degredation

IAD Network Incident