Jul <var data-var='date'>13</var>, <var data-var='time'>18:41</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>13</var>, <var data-var='time'>16:39</var> UTC Monitoring - We have some out of hours house-keeping to do here but this incident is otherwise resolved. We will be monitoring for a few hours before closing to be certain. Jul <var data-var='date'>13</var>, <var data-var='time'>16:04</var> UTC Update - CDRs are now being processed at full speed and catching up rapidly. Jul <var data-var='date'>13</var>, <var data-var='time'>15:13</var> UTC Update - Portal and API (v4) are now fully operational. CDRs remain lagged. Jul <var data-var='date'>13</var>, <var data-var='time'>14:29</var> UTC Identified - We have lost hardware in Slough. Voice services have failed over as expected and are unaffected. Most other services have recovered too although we are currently dealing with some Redis master failures elsewhere on the network. This will manifest in portal logins and writing to selected API end-points. Separately CDR processing is several million delayed so API balances etc. will not be current. We are recovering Redis clusters as P1 and will advise progress.

Slough hardware outage

Jun <var data-var='date'>15</var>, <var data-var='time'>15:17</var> UTC Resolved - The engineers attended an intermediate site along the affected route and found multiple fibre pairs had been damaged at the ODF there. These were replaced, testing has been completed, and the link has been restored with all alarms clear. We will continue to monitor. Jun <var data-var='date'>15</var>, <var data-var='time'>11:34</var> UTC Update - Our transport provider has identified the fault location and fibre engineers are due to the location at 13:30 to begin repairs. Jun <var data-var='date'>15</var>, <var data-var='time'>10:17</var> UTC Identified - Reduced network redundancy in our Slough AZ. An optical link back to central London is currently down, and traffic is being routed via alternative paths while we work with the transport provider to restore the link. AZ considered at risk.

Reduced network redundancy in Slough availability zone

Mar <var data-var='date'> 5</var>, <var data-var='time'>04:24</var> UTC Completed - The maintenance has taken longer than expected, but is now complete and the affected fibre leg has been fully restored into service. We will continue to monitor. Mar <var data-var='date'> 4</var>, <var data-var='time'>20:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'> 4</var>, <var data-var='time'>17:57</var> UTC Scheduled - Low light levels were detected today on one of our London fibre legs between Telehouse North and Volta (Verne, Great Sutton Street), so our fibre provider will carry out inspection and precautionary maintenance this evening, starting at 20:00 GMT. No outage has been observed and the link has remained operational. As this work will require brief disconnects on that leg, we have moved traffic away from the affected path to avoid unnecessary flaps. The rest of the London ring and other redundant paths will remain in service throughout, so no adverse impact is expected. We will continue to monitor and share further updates if anything changes.

Short-notice Maintenance on London Fibre Ring

Dec <var data-var='date'>24</var>, <var data-var='time'>16:01</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'>24</var>, <var data-var='time'>12:07</var> UTC Monitoring - The affected hardware in our Manchester availability zone has been restored and reintroduced to the media pool. We are monitoring performance closely before marking this incident as fully resolved. Dec <var data-var='date'>24</var>, <var data-var='time'>10:40</var> UTC Identified - We are currently operating with reduced media capacity in our Manchester availability zone following a suspected hardware failure. There is no impact to services and calls are completing normally. Engineers are working to restore the affected hardware and capacity will be returned to normal in due course. In the meantime, our network is designed so that any single availability zone can carry the entire network load if all others fail. This update is for transparency only and no customer action is required.

Reduced media capacity in Manchester AZ

Oct <var data-var='date'>14</var>, <var data-var='time'>11:34</var> UTC Completed - The planned maintenance at Interxion has been completed. The works have identified the need for additional remedial work, which will be carried out during future maintenance windows. Further maintenance windows will be posted once these are scheduled. Oct <var data-var='date'>14</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>10</var>, <var data-var='time'>10:56</var> UTC Scheduled - A third party will be performing maintenance on our London fibre ring between 09:00 and 11:00 UTC (10:00 - 12:00 BST) on Tuesday 14th. This work involves cleaning and testing fibre connectors at the IXN (Interxion) site, which will temporarily affect the fibre leg between Volta (Great Sutton Street) and Telehouse North. To avoid unnecessary route flapping, the affected link will be taken out of service in advance and restored once the work is complete. The other direction in our London ring, as well as other redundant paths, will remain operational throughout. Connectivity will therefore be considered at risk but expected to remain available.

Planned Fibre Ring Maintenance in London

Oct <var data-var='date'>14</var>, <var data-var='time'>09:05</var> UTC Resolved - Colt have accepted our outstanding ports so this incident will be resolved after nearly 2 months. Oct <var data-var='date'> 7</var>, <var data-var='time'>10:51</var> UTC Update - Colt claim to be doing excellently with processing sales orders and say "voice services will start to come back online from this week". We're not sure if this means honouring their regulatory obligations after 50 days, or just sales activity. We will advise if/when they resume porting or give a further update. Oct <var data-var='date'> 2</var>, <var data-var='time'>09:30</var> UTC Update - In a recent update to their customers Colt claim to be nearly able to deliver on some sales orders but their regulatory obligations in porting are apparently a lower priority - no porting still. Sep <var data-var='date'>16</var>, <var data-var='time'>11:11</var> UTC Update - Colt are still not porting and have not updated us as to the status. They have updated their "cyber-incident" page (which isn't linked on their website and is set to prevent indexing by search engines) to say they are committed to transparency. We also hear they have advised some Enterprise customers they will be unable to deliver new connectivity orders until 2026. We cannot confirm this, but if true it would be reasonable to assume that porting numbers out is not of a higher priority to them than new business. We know Ofcom are aware and hope they will be asserting appropriate priorities and full disclosure. Sep <var data-var='date'> 8</var>, <var data-var='time'>08:22</var> UTC Update - 27 days since Colt responded to their cyber-security incident which, after eventually admitting they have consistently down-played, they continue to be unable to process number ports. Please continue submitting them as we are queuing this side. Sep <var data-var='date'> 2</var>, <var data-var='time'>09:25</var> UTC Update - Colt continue to not process number ports despite down-playing the cyber-security incident which they initially responded to 21 days ago. Aug <var data-var='date'>28</var>, <var data-var='time'>12:24</var> UTC Update - Colt are keen to emphasise on their status page (https://www.colt.net/status/) how this didn't affect customer systems. They have also marked their cyber-incident page at https://www.colt.net/go/cyber-incident/ to "noindex" to hide it from search engines, and it isn't linked anywhere on their website. None of them mention number portability either, which they continue to not do, despite us now being in day 16. Aug <var data-var='date'>27</var>, <var data-var='time'>07:54</var> UTC Update - Colt have contacted us to say that "all port-in and port-out activities are currently on hold until next week, as we’re still working internally to stabilize the situation" Aug <var data-var='date'>26</var>, <var data-var='time'>10:10</var> UTC Update - Colt are still not processing number ports. Please continue to submit porting orders normally, we have a queue and are chasing them daily. Aug <var data-var='date'>24</var>, <var data-var='time'>11:42</var> UTC Update - An excellent blog has been published documenting the events so far at https://doublepulsar.com/colt-technical-services-gets-ransomwared-via-sharepoint-initial-access-some-learning-points-617da7e27ebc Colt are still not processing number ports. Aug <var data-var='date'>22</var>, <var data-var='time'>11:58</var> UTC Update - Other than restating their status page to say this was a cybersecurity incident from day one (formerly an "IT issue"), and their compromised documents being mid-auction, there is no update here. Their porting desk appear to have stopped the daily updates and are still not processing orders. Aug <var data-var='date'>19</var>, <var data-var='time'>12:51</var> UTC Update - There is no update here other than Colt's porting team have advised there will be no imports or exports "until further notice". We continue to queue POV and porting orders for them. Aug <var data-var='date'>18</var>, <var data-var='time'>14:23</var> UTC Update - Colt are still unable to process number ports, we assume as a consequence of their well publicised cyber-attack. A list of compromised files, which is asserted to be the files compromised by the attacks is at (https://www.klos.com/~john/colt_filename_tree.txt) This list contains a number of interoperability documents between Simwood and Colt. The list also contains documents that appear to be Number Portability Order Forms, although it is unclear as to whether these relate to Simwood customers. NPORs will contain some information about the Subscriber changing providers, e.g. name, address, and telephone numbers. Additionally, there is other information clearly of a sensitive nature on the list, but that does not, as far as we can tell, relate to Simwood and its customer’s relationships. At this time, we understand that Colt are working to prevent further disclosure, however, this is all the detail we have at this time. Our customers should engage their Data Protection Officers regarding the consequences of the attack on Colt as a matter of urgency. Aug <var data-var='date'>15</var>, <var data-var='time'>16:45</var> UTC Update - There is no news to report here although the now admitted attack is being actively reported on, e.g. https://www.bleepingcomputer.com/news/security/colt-telecom-attack-claimed-by-warlock-ransomware-data-up-for-sale/ Aug <var data-var='date'>14</var>, <var data-var='time'>17:06</var> UTC Update - Colt have now confirmed an ongoing cyber-incident (https://www.colt.net/status/). At the moment voice traffic on existing ported numbers seems unaffected. Aug <var data-var='date'>14</var>, <var data-var='time'>13:55</var> UTC Monitoring - Colt have advised that they are unable to complete any import or export porting orders until further notice. Their Status page (https://www.colt.net/status/) suggests an IT issue while online speculation (https://cyberplace.social/@GossiTheDog/115022343170487477) is of an undisclosed cyber attack. We will advise when we have further news.

Porting from Colt

Oct <var data-var='date'> 6</var>, <var data-var='time'>11:06</var> UTC Resolved - Bank details are updated on the document linked from https://si.mw/jp2qgym (our support portal). Please note this is on the simwood.com domain - we will never email you bank details. Alternative payment methods (card or stablecoin) are described at: https://si.mw/ro4n9hd We're sorry for any inconvenience caused by AirWallex's grossly unreasonable and still unjustified actions. Oct <var data-var='date'> 6</var>, <var data-var='time'>09:53</var> UTC Monitoring - New payment processing is now in place in all currencies and account details are being updated now. Customers may continue to send to the Lloyds account but payments will be manually processed; sending to the new bank details (with only the account number as reference) will resume automated top-ups on receipt. AirWallex have refused to provide any information and continue to reject incoming payments. We will therefore be terminating all AirWallex accounts across all businesses, regardless of what this transpires to be, and updating details in due course. Oct <var data-var='date'> 3</var>, <var data-var='time'>09:07</var> UTC Investigating - AirWallex, who we use for global automated payment processing, have "temporarily" suspended our account for incoming payments. They haven't said why or for how long and gave us the courtesy of zero notice. Any payments sent this route will therefore be returned to senders and not applied to Simwood accounts. Please use our back-up Lloyds Bank account which is shown in the document linked from https://si.mw/jp2qgym or one of the alternative payment methods (card or stablecoin) described at: https://si.mw/ro4n9hd . We apologise for any inconvenience and will update this as soon as we know more about this as yet unjustified (and in our opinion totally undeserved and heavy-handed) action.

Banking outage

Aug <var data-var='date'> 1</var>, <var data-var='time'>12:28</var> UTC Resolved - This incident is now fully resolved and appears to have been a fault on the Simwood network. A minor change relating to the validation of caller ID against Ofcom data was rolled around the network from 11am. It had passed all unit tests and presented no errors in production. However, it had the consequence of us assuming at the time of calls being routed that our costs included maximum origin surcharges on destinations that apply them, i.e. UK mobile networks except O2 and certain fixed line operators. While we routinely route calls at a loss and do not pass on origin surcharges, the consequence of this was that customer profiles became progressively loss-making for these destinations and overall. Our fraud and anti-arbitrage protections then began blocking such ostensibly loss-making traffic for customers who had tripped their account or destination specific loss-limits. Surcharges vary between 34ppm and 200ppm depending on network meaning account traffic profile would have affected if and when limits were reached. Some did not trip at all, others would have progressively tripped after 11.08, resulting in some calls to affected destinations being blocked. Calls to destinations without surcharges were unaffected as were those for customers who had not tripped particular limits; nevertheless it resulted in a progressive reduction in our traffic to certain networks over the ensuing hour. Our first support ticket was at 11:42, with a greater volume of reports from 12:00 and mitigation(s) fully deployed by 12:13. There were two mitigations: 1) customer loss-limits were raised at 12:11 and 2) the deployed image was fully rolled back around the network from 12:10 to 12:13. The last call exhibiting this behaviour, albeit not blocked, was 12:13. We identified and corrected the bug introduced in logic and are adding further unit tests to ensure such an occurrence cannot repeat. Once the new build fully passes the new tests, this updated image will be carefully deployed later today. We have also added specific global margin monitoring to alert very quickly on unnaturally changes in apparent profitability. Apologies for this issue which, as usual, we own and will learn from. Aug <var data-var='date'> 1</var>, <var data-var='time'>11:46</var> UTC Monitoring - We believe this was mitigated at 12.13 - any examples after that please let us know. We believe we understand what the issue was but continue to investigate and will report fully shortly. To correct the previous report, this would have affected calls to UK destinations applying an origin surcharge - mobile networks are the most egregious but some landlines would have been affected too. Aug <var data-var='date'> 1</var>, <var data-var='time'>11:23</var> UTC Identified - We were seeing reduced volume to certain UK mobile networks which we believe we have mitigated. We are investigating why this might be with the last report being at 12.06 BST.

Mobile call failures

Jul <var data-var='date'>24</var>, <var data-var='time'>21:23</var> UTC Resolved - Following the initial reports of issues calling certain mobile numbers, we’ve been monitoring closely and can confirm once again that there have been no faults or abnormalities on the Simwood network throughout the period. While some customers may still experience issues, these appear to relate to factors outside of our network. We encourage those affected to consult general media sources for the latest updates on wider network conditions. Thank you for your understanding. Jul <var data-var='date'>24</var>, <var data-var='time'>15:16</var> UTC Investigating - We’ve received several reports of calls failing to certain mobile numbers. Our systems are operating normally, and we’ve found no issues within the Simwood network. There are indications the issue may lie with another network, and we’re actively monitoring the situation while we investigate further.

Reported issues calling some mobile destinations

Jun <var data-var='date'>21</var>, <var data-var='time'>09:43</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>20</var>, <var data-var='time'>13:22</var> UTC Monitoring - Cogent have confirmed issues on their network. Our sessions with them remain shut down to prevent impact. We will continue monitoring and will only reintroduce routes for Cogent in Slough once we are confident their network is stable. Jun <var data-var='date'>20</var>, <var data-var='time'>12:41</var> UTC Investigating - Customers connecting to our services in Slough over Cogent transit may have seen instability over the last 15 minutes or so. We have temporarily shutdown sessions with Cogent to alleviate any issues. Those on-net or reaching us over peering sessions (which is the majority) and other ultimate transit providers, or reaching other availability zones over Cogent have been unaffected. We strongly encourage all customers to connect into us directly wherever possible.

Routing instability via Cogent in Slough

May <var data-var='date'>29</var>, <var data-var='time'>12:15</var> UTC Resolved - The span has been restored and the links have been brought back into service, restoring full redundancy around our London ring. May <var data-var='date'>29</var>, <var data-var='time'>10:10</var> UTC Investigating - We are seeing one of our physical fibre spans between Telehouse North and IXN hard down currently, suggesting physical disturbance. We have preemptively shut down links using this whilst it is investigated and repaired. This reduces redundancy around our London ring but as we have multiple other paths, no service impact is expected.

London: Reduced fibre network redundancy

May <var data-var='date'>11</var>, <var data-var='time'>06:44</var> UTC Resolved - This incident has been resolved. May <var data-var='date'>10</var>, <var data-var='time'>08:00</var> UTC Identified - The Slough Availability Zone is currently operating on a single power feed due to planned power infrastructure maintenance by our datacenter provider. While all services remain fully operational, the zone should be considered at risk until full power redundancy is restored. The maintenance window is scheduled to conclude by 22:59 UTC on Saturday, 10 May. We are closely monitoring the situation and will provide updates as needed.

Slough AZ – At-Risk Advisory (Power Maintenance)

Dec <var data-var='date'>20</var>, <var data-var='time'>05:00</var> UTC Completed - The scheduled maintenance has been completed. Dec <var data-var='date'>20</var>, <var data-var='time'>01:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>19</var>, <var data-var='time'>17:48</var> UTC Scheduled - We will be performing planned maintenance on our London fibre ring between 01:00 and 05:00 UTC, during which an intermediate regeneration point will be removed. This will temporarily affect the fibre link between our London Telehouse East and Volta (Great Sutton Street) site until it can be re-spliced. While other redundant routes will remain available between these sites, connectivity will be temporarily reduced and therefore considered at risk. This maintenance was originally scheduled for 19th, but wasn’t completed during the maintenance window, so has been rescheduled for the 20th.

Dec <var data-var='date'>19</var>, <var data-var='time'>05:00</var> UTC Completed - The scheduled maintenance has been completed. Dec <var data-var='date'>19</var>, <var data-var='time'>01:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>17</var>, <var data-var='time'>16:23</var> UTC Scheduled - We will be performing planned maintenance on our London fibre ring between 01:00 and 05:00 UTC, during which an intermediate regeneration point will be removed. This will temporarily affect the fibre link between our London Telehouse East and Volta (Great Sutton Street) site until it can be re-spliced. While other redundant routes will remain available between these sites, connectivity will be temporarily reduced and therefore considered at risk.

Sep <var data-var='date'>25</var>, <var data-var='time'>22:42</var> UTC Resolved - All alarms have been cleared. Resolving this incident. Sep <var data-var='date'>25</var>, <var data-var='time'>22:17</var> UTC Monitoring - The optical card has been replaced and the link has been restored. Status will remain in “monitoring” until we have confirmation from the engineers, vendor, and when we have satisfied our internal checks. Sep <var data-var='date'>25</var>, <var data-var='time'>16:56</var> UTC Investigating - There is currently reduced network redundancy in our Manchester availability zone due to an identified issue with an optical transport link. Vendor has identified an issue with a line card in Leeds and an engineer is being sent to perform a replacement, expected on-site at 22:15. No service or traffic impacted, but site is considered at-risk.

Reduced network redundancy in Manchester availability zone

Apr <var data-var='date'>30</var>, <var data-var='time'>16:24</var> UTC Resolved - Volta has been returned to N+3 connectivity and Telehouse East reconnected at N+1. Voice service was unaffected by this incident but full redundancy in the affected of our 3 UK Availability Zones has been restored. Apr <var data-var='date'>30</var>, <var data-var='time'>15:55</var> UTC Update - The datacentre have confirmed that these two fibre pairs were indeed seperately cut within the building and are reviewing repair options. Apr <var data-var='date'>30</var>, <var data-var='time'>14:53</var> UTC Identified - We noticed some instability on the network around London this afternoon which resulted from two of our fibre pairs out of Volta (both East and West loops) having been disconnected/cut within the building within 5 seconds of each other. This is being investigated but the result is that Volta is reduced to N+1 from N+3 redundancy, and Telehouse East (which has no voice services whatsoever) has been temporarily isolated. The network continues to operate at 100% otherwise with no interruption to voice or related services but further events are always possible. We do not expect a rapid resolution to the cut fibre but will update when there is one.

At risk - London Availability Zone

Nov <var data-var='date'>14</var>, <var data-var='time'>16:00</var> UTC Resolved - At around 15.45 this afternoon we were made aware of inbound calls to some number ranges hosted on our network, in some circumstances, being delivered to customers with the destination number in the RURI and To header truncated, resulting in those calls failing to connect due to the target number not being recognised. Within 5 minutes we had identified the issue was related to an update we were in the process of rolling out, which was intended to work around an increasing number of calls being sent to us by BT with invalid destination numbers. This was necessary because they were unwilling/incapable of fixing in their own call routing, with the numbers matching their routing plan despite being invalid. By 15.55 we had rolled back the update across all availability zones, and affected customers had confirmed calls were once again being delivered as expected. We are currently working through examples provided to understand why each specific routing case was impacted and will update here with our findings in due course.

Inbound calls to some hosted ranges failing

Feb <var data-var='date'>22</var>, <var data-var='time'>15:00</var> UTC Resolved - At 14h44 we were notified of the failure of a primary database node in London (Volta). This is a planned failure scenario and as designed service failed over cleanly to a candidate replacement in Slough (LD4). At 14h50 our call monitoring reported increased PDD (Post Dial Delay) from some parts of the network. This was owing to several call-routing nodes which were previously slaves to the failed master resyncing, and thus being unavailable for service. In this scenario, call-routing fails over to other back-up instances, which it did. Depending on the precise local state at the time of the call this can increase PDD. The first node had resynced by 14h58 and by 15h03 the last node had fully resynced and PDD had returned to normal levels everywhere. Our monitoring shows that less than 15% of calls network wide were impacted by increased PDD but customer experiences may vary according to their own timers and failover protocols. We are however investigating utilisation of the backup routing instances which, whilst not experience affecting, was not as evenly distributed as designed.

Elevated PDD

Dec <var data-var='date'>12</var>, <var data-var='time'>09:00</var> UTC Resolved - We have identified delays in received SMS delivery.

Delayed inbound SMS

Dec <var data-var='date'>10</var>, <var data-var='time'>22:01</var> UTC Completed - The scheduled maintenance has been completed. Dec <var data-var='date'>10</var>, <var data-var='time'>20:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'> 9</var>, <var data-var='time'>19:23</var> UTC Scheduled - We will be performing a reboot of one of our container hosts in Slough as part of some scheduled maintenance tomorrow evening. We do not anticipate any impact to services but will be shifting traffic away from this availability zone for the duration of the maintenance window as a precaution. Site specific IP addresses will failover between nodes but may see brief windows of inaccessibility. DNS will be updated to direct traffic to other sites for customers configured in accordance with our interop. Our interop specifies that customers should not send directly to an IP address and, if doing so is unavoidable or for those with wrongly cached DNS, customers should configure their own fail-over to other sites. Anycast services will be served from the next closest node but routing will change. This maintenance should therefore not be service affecting but the network is considered at risk through a reduced level of redundancy during it.

Slough container host maintenance

Sep <var data-var='date'>16</var>, <var data-var='time'>00:03</var> UTC Completed - The scheduled maintenance has been completed. Sep <var data-var='date'>15</var>, <var data-var='time'>21:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>15</var>, <var data-var='time'>14:08</var> UTC Scheduled - We will be performing maintenance on one of our container hosts in Slough as part of planned maintenance this evening. We do not anticipate any impact to services but will be shifting traffic away from this availability zone for the duration of the maintenance window. Site specific IP addresses will failover between nodes but may see brief windows of inaccessibility. DNS will be updated to direct traffic to other sites for customers configured in accordance with our interop. Our interop specifies that customers should not send directly to an IP address and, if doing so is unavoidable or for those with wrongly cached DNS, customers should configure their own fail-over to other sites. Anycast services will be served from the next closest node but routing will change. This maintenance should therefore not be service affecting but the network is considered at risk through a reduced level of redundancy during it.

Jun <var data-var='date'> 7</var>, <var data-var='time'>17:38</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'> 7</var>, <var data-var='time'>16:50</var> UTC Monitoring - This has been stable since the incident was opened but we continue to investigate the root cause and will likely need to continue to do so once the incident is closed. DNS has been returned but we again urge customers to respect SRV or at least DNS for seamless failover in cases like this. Jun <var data-var='date'> 7</var>, <var data-var='time'>15:48</var> UTC Investigating - We are seeing elevated errors on our London SIP edge. Whilst this will not affect customers properly configured to use FQDN and SRV, we have manually failed DNS over to another Availability Zone for those who are not. We are investigating the underlying issue.

London SIP edge

May <var data-var='date'>24</var>, <var data-var='time'>09:00</var> UTC Resolved - At 10.01 we lost reachability to equipment in Manchester over certain routes and a loss of power in Equinix Kilburn became apparent. Whilst power and service were restored a few minutes later we understand the site is running on generators and a UPS fault has been identified. We have very little active equipment in Kilburn but it is a major hub for networks and further power issues will affect reachability; it should be considered at risk.

Manchester power issues

Apr <var data-var='date'>28</var>, <var data-var='time'>20:57</var> UTC Resolved - It appears that a routine upgrade to our call routing engine disrupted some incoming calls this evening from 20:25 to 20:57. This type of update is quite normal for us and happens very often as part of our continuous development and deployment. It was being progressively rolled around the network such that calls could failover to the previous version in the event of any failure. However it did not fail and whilst passing all unit tests appears to have caused unexpected call rejections for calls hitting it. The deployment was paused as soon as we were aware of issues and has been rolled back. We will investigate the underlying issue and resolve before resuming continuous deployments. Apologies for anyone affected.

Brief failures on inbound calling

Feb <var data-var='date'>26</var>, <var data-var='time'>14:44</var> UTC Resolved - All services have been fully restored in Volta. Feb <var data-var='date'>26</var>, <var data-var='time'>11:53</var> UTC Monitoring - Portal access has been fixed for all customers and we continue to monitor all services Feb <var data-var='date'>26</var>, <var data-var='time'>11:51</var> UTC Identified - Whilst continuing to monitor the earlier problem we are aware that the portal is unavailable for some customers and are addressing that issue Feb <var data-var='date'>26</var>, <var data-var='time'>10:43</var> UTC Monitoring - We have been checking services and are confident all is working normally. We will be monitoring this until power is fully restored and services migrated back. Feb <var data-var='date'>26</var>, <var data-var='time'>09:29</var> UTC Identified - We have lost a power feed in Volta which has reduced redundancy across the whole site and taken some servers off-line. Affected services have already automatically migrated to other sites for those following our standard configurations and DNS has been modified for those forcing traffic to this particular site. Customers who haven't followed the interop at all (and are forcing traffic by IP address) will need to manually update their config. We are monitoring the situation and will advise on recovery in due course.

Is Simwood down?

Simwood status is UP

Lastest published events

Slough hardware outage

Reduced network redundancy in Slough availability zone

Short-notice Maintenance on London Fibre Ring

Simwood Services, Components and Regions

Recent Simwood events

Slough hardware outage

Reduced network redundancy in Slough availability zone

Short-notice Maintenance on London Fibre Ring

Reduced media capacity in Manchester AZ

Planned Fibre Ring Maintenance in London

Porting from Colt