THIS IS A SCHEDULED EVENT Apr <var data-var='date'>16</var>, <var data-var='time'>17:00</var> - <var data-var='time'>21:00</var> UTCApr <var data-var='date'> 5</var>, <var data-var='time'>15:09</var> UTC Scheduled - As part of our network modernization and stability initiative, we are moving the Subnet Manager to new hardware. This will improve the service's resilience and performance.

Stability Improvements: Subnet Manager Migration in TXL (2/2)

THIS IS A SCHEDULED EVENT Apr <var data-var='date'>9</var>, <var data-var='time'>17:00</var> - <var data-var='time'>21:00</var> UTCApr <var data-var='date'> 5</var>, <var data-var='time'>15:06</var> UTC Scheduled - As part of our network modernization and stability initiative, we are moving the Subnet Manager to new hardware. This will improve the service's resilience and performance.

Stability Improvements: Subnet Manager Migration in TXL (1/2)

Apr <var data-var='date'> 8</var>, <var data-var='time'>08:30</var> UTC Completed - The scheduled maintenance has been completed.Apr <var data-var='date'> 8</var>, <var data-var='time'>07:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>28</var>, <var data-var='time'>08:16</var> UTC Scheduled - We would like to inform you that regular maintenance of the Cloud Panel is due to be carried out. During this time, access to the Cloud Panel will be limited while access to the vSphere Clients, NSX Managers and the management VPN themselves will not be impacted. Availability and accessibility of your Private Cloud resources will remain unaffected.

Cloud Panel Maintenance

Apr <var data-var='date'> 8</var>, <var data-var='time'>15:29</var> UTC Update - The second configuration update rollout is currently in progress, and we have confirmed initial improvements related to CPU performance. Due to the size of the fleet, we expect the rollout to take some time to complete. Throughout the process, customers will see performance gains as soon as the specific hosts supporting their workloads have been updated. We will provide a final update once the rollout is finished.Apr <var data-var='date'> 8</var>, <var data-var='time'>11:28</var> UTC Monitoring - The adjustment was rolled out. Our Compute team is seeing dropping CPU steal time. We are monitoring the situation. Our Tech Teams are preparing another rollout that should improve the performance further.Apr <var data-var='date'> 8</var>, <var data-var='time'>10:12</var> UTC Update - Our compute team has successfully tested the proposed fix for the CPU core affinity and is preparing a rollout. We will monitor the results.Apr <var data-var='date'> 8</var>, <var data-var='time'>09:54</var> UTC Update - Our compute team has found another factor negatively impacting CPU performance for affected VMs. We are currently testing a potential transparent resolution for the problematic CPU affinity setting.Apr <var data-var='date'> 8</var>, <var data-var='time'>08:27</var> UTC Identified - We have identified an increase in CPU steal time on affected hosts. Our Compute team has identified a likely culprit and is currently testing a potential mitigation to ensure its effectiveness before a rollout.Apr <var data-var='date'> 8</var>, <var data-var='time'>06:48</var> UTC Investigating - We are currently investigating performance degradation affecting compute components in our FRA DC. This issue is impacting a subset of Virtual Machines (VMs) and Kubernetes Clusters. We will provide further updates as our investigation progresses.

Performance Degradation Compute FRA

Apr <var data-var='date'> 1</var>, <var data-var='time'>20:18</var> UTC Resolved - We are marking the incident as resolved, as no further anomalies were detected. The root cause was a transient latency issue in one of the redundant CoreDNS pods, which led to kube-apiservers being unable to discover etcd instances in time. We are currently developing better mitigation options for the excessive NXDOMAIN requests observed during the incident.Mar <var data-var='date'>31</var>, <var data-var='time'>13:10</var> UTC Monitoring - All affected Control Planes have recovered. We are currently monitoring the affected clusters for further anomalies.Mar <var data-var='date'>31</var>, <var data-var='time'>12:22</var> UTC Investigating - We are currently investigating service degradation for Control Planes of some Kubernetes clusters in the FRA DC.

Managed Kubernetes: Service Degradation for Control Planes in FRA

Mar <var data-var='date'>30</var>, <var data-var='time'>22:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>30</var>, <var data-var='time'>18:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>10</var>, <var data-var='time'>07:58</var> UTC Scheduled - We would like to inform you that regular maintenance on our IONOS cloud backup service is due to be carried out. The backup console will not be available during the maintenance, scheduled backups will be delayed.

Maintenance of the Backup service

Mar <var data-var='date'>30</var>, <var data-var='time'>15:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>30</var>, <var data-var='time'>12:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>30</var>, <var data-var='time'>11:05</var> UTC Scheduled - Due to an urgent system requirement, we are performing unscheduled maintenance on our provisioning infrastructure. While we strive to keep any disruption to an absolute minimum, provisioning jobs may be temporarily unavailable or experience slight delays during this window. There will be no impact on existing, already provisioned resources.

Unscheduled Maintenance - Provisioning

Mar <var data-var='date'>26</var>, <var data-var='time'>22:30</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>26</var>, <var data-var='time'>18:30</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>10</var>, <var data-var='time'>07:41</var> UTC Scheduled - We would like to inform you that regular maintenance on our IONOS cloud backup service is due to be carried out. The backup console will not be available during the maintenance, scheduled backups will be delayed.

Mar <var data-var='date'>25</var>, <var data-var='time'>11:10</var> UTC Resolved - This incident is resolved. We will publish a root cause analysis once it is compiled.Mar <var data-var='date'>25</var>, <var data-var='time'>03:00</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'>25</var>, <var data-var='time'>00:43</var> UTC Investigating - We are currently investigating limitations in the availability of the Cloud-API and the data center designer (DCD). We will provide further information as soon as possible.

Availability of API and DCD limited

Mar <var data-var='date'>23</var>, <var data-var='time'>21:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>23</var>, <var data-var='time'>16:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>20</var>, <var data-var='time'>16:18</var> UTC Scheduled - We would like to inform you that an important maintenance of the Managed Kubernetes is due to be carried out on Monday evening. During the maintenance, connections to your cluster's kube-apiserver might occasionally become slow or unavailable, possibly also requiring a retry. Please be aware that during the maintenance, Managed Network Resources cannot be changed or created. Kubernetes resources could also be affected, meaning you might not be able to make changes to your clusters or nodepools, either via the DCD or the API.

Managed Kubernetes Maintenance

Mar <var data-var='date'>19</var>, <var data-var='time'>13:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>19</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>18</var>, <var data-var='time'>15:44</var> UTC Scheduled - We will be performing a scheduled upgrade of the network management software (SDN) for our server infrastructure. This maintenance will bring our systems to the latest stable version to ensure optimal performance. This maintenance might cause an interruption of network connectivity for up to 1 (one) minute per processed host server. If you have virtual machines or services that are hosted on various hosts, you might experience multiple instances of this service disruption.

SDN Version Upgrade

Mar <var data-var='date'>18</var>, <var data-var='time'>22:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>18</var>, <var data-var='time'>20:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>18</var>, <var data-var='time'>16:05</var> UTC Scheduled - We need to replace a faulty switch to ensure redundancy and stability of the network. There is no impact expected for customers.

Emergency Maintenance: TXL - Switch Replacement

Mar <var data-var='date'>18</var>, <var data-var='time'>07:30</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>18</var>, <var data-var='time'>06:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>14</var>, <var data-var='time'>08:03</var> UTC Scheduled - We would like to inform you that regular maintenance of the Cloud Panel is due to be carried out. During this time, access to the Cloud Panel will be limited while access to the vSphere Clients, NSX Managers and the management VPN themselves will not be impacted. Availability and accessibility of your Private Cloud resources will remain unaffected.

Mar <var data-var='date'>17</var>, <var data-var='time'>17:04</var> UTC Resolved - This incident is now marked resolved, as all affected control planes have returned to and stayed in a stable state. A Root Cause Analysis (RCA) is underway and will be published here once finalized.Mar <var data-var='date'>16</var>, <var data-var='time'>19:59</var> UTC Monitoring - Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progressMar <var data-var='date'>16</var>, <var data-var='time'>19:58</var> UTC Update - Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progressMar <var data-var='date'>16</var>, <var data-var='time'>19:34</var> UTC Update - The changes applied to some control plane clusters have had positive effects. The team is continuing the rollout to other affected clusters.Mar <var data-var='date'>16</var>, <var data-var='time'>13:56</var> UTC Update - We are marking DBaaS as recovered. Our Kubernetes Team is currently working on stabilizing the Kubernetes Control Plane. We are focusing on mitigating recurring load spikes influencing stability.Mar <var data-var='date'>16</var>, <var data-var='time'>12:57</var> UTC Update - We are marking the Container Registry Service as recovered.Mar <var data-var='date'>16</var>, <var data-var='time'>12:22</var> UTC Update - We are closing the incident for the AI Model Hub. All metrics have recovered and the service should be up and running again normally.Mar <var data-var='date'>16</var>, <var data-var='time'>11:57</var> UTC Update - We are adding the Container Registry as an affected Service. Customers may currently experience issues pulling and pushing images from the Registry.Mar <var data-var='date'>16</var>, <var data-var='time'>11:37</var> UTC Update - Our Kubernetes Team has deployed a fix for the affected AI Model Hub Database Services. We currently see metrics improving and monitoring the situation closely.Mar <var data-var='date'>16</var>, <var data-var='time'>11:05</var> UTC Update - We are expanding the scope of this incident to include DBaaS and AI Model Hub. We have observed an increased error count originating from PostgresDB on Kubernetes. Additionally, to improve transparency, the previously reported separate incident regarding the AI Model Hub (https://status.ionos.cloud/incidents/rmgs845klm32) is being merged into this primary incident.Mar <var data-var='date'>16</var>, <var data-var='time'>10:04</var> UTC Identified - The team has identified the root cause as a resource constraint within the etcd database. Mitigation efforts are currently underway.Mar <var data-var='date'>16</var>, <var data-var='time'>08:24</var> UTC Investigating - Some customers may experience connection problems to the control plane and degraded functionality of kubernetes. Our teams are investigating and working on a resolution.

MK8s: Partial Connectivity Degradation to Control Planes

Mar <var data-var='date'>16</var>, <var data-var='time'>11:08</var> UTC Resolved - To improve visibility and streamline communication, we are merging this incident into the MK8s incident (tracked here: https://status.ionos.cloud/incidents/h9x5s66m4r28). Consequently, we will close this specific entry and provide all future updates via the referenced incident link.Mar <var data-var='date'>16</var>, <var data-var='time'>10:01</var> UTC Identified - The team has traced the root cause to the ongoing Kubernetes incident (link). Both teams are currently working to restore service. (https://status.ionos.cloud/incidents/h9x5s66m4r28)Mar <var data-var='date'>16</var>, <var data-var='time'>08:28</var> UTC Investigating - Our AI Model Hub Team is currently investigating increased error rates in the Embeddings functionality in the AI Model Hub.

AI Model Hub - Increased Error Rate in Embeddings

Mar <var data-var='date'>12</var>, <var data-var='time'>22:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>12</var>, <var data-var='time'>21:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>12</var>, <var data-var='time'>14:49</var> UTC Scheduled - We are implementing urgent changes to enhance the stability and performance of the TXL network. In rare instances, customers may experience brief, intermittent connectivity interruptions.

Emergency Maintenance: TXL Network

Mar <var data-var='date'>12</var>, <var data-var='time'>20:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>12</var>, <var data-var='time'>19:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>12</var>, <var data-var='time'>14:46</var> UTC Scheduled - We are implementing urgent changes to enhance the stability and performance of the PAR network. We do not expect customer facing service degradation.

Emergency Maintenance: PAR Network

Mar <var data-var='date'>12</var>, <var data-var='time'>13:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'>12</var>, <var data-var='time'>12:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>12</var>, <var data-var='time'>11:47</var> UTC Scheduled - We are implementing urgent changes to enhance the stability and performance of the LAS network. In rare instances, customers may experience brief, intermittent connectivity interruptions.

Emergency Maintenance: LAS Network

Mar <var data-var='date'>11</var>, <var data-var='time'>21:19</var> UTC Resolved - We are marking this incident as resolved. Our Network Team will publish an RCA here, once it is compiled.Mar <var data-var='date'>11</var>, <var data-var='time'>19:27</var> UTC Monitoring - We are placing the incident in a monitoring state. Our Network Team is closely monitoring the cluster and working to restore full redundancy.Mar <var data-var='date'>11</var>, <var data-var='time'>18:54</var> UTC Update - The deployed change has had positive effect. We are downgrading the impact level while continuing to monitor the cluster closely.Mar <var data-var='date'>11</var>, <var data-var='time'>18:31</var> UTC Identified - In response to monitoring alerts, our network team deployed a change to stabilize the network in the affected cluster. We will post another update at 19:00 UTC—or as soon as new information becomes available.Mar <var data-var='date'>11</var>, <var data-var='time'>18:13</var> UTC Investigating - We are currently investigating network connectivity issues in our TXL datacenter.

Network Connectivity Issues in TXL

Mar <var data-var='date'>11</var>, <var data-var='time'>19:20</var> UTC Resolved - We are marking this incident as resolved. The incident was caused by capacity constraints following a hardware failure. While capacity has been restored, we still see some usage‑specific constraints with the Llama 3.1 405B Instruct model. Our AI ModelHub team will deploy optimizations to the model to increase performance and reliability. We recommend that users still experiencing issues with the model check GPT‑OSS 120B as a potential (temporary) replacement.Mar <var data-var='date'> 9</var>, <var data-var='time'>18:53</var> UTC Monitoring - Our AI Model Hub Team has mitigated the incident. While the underlying root cause is not yet fully established or resolved, the model service should be stable. We are monitoring the situation while the investigation is ongoingMar <var data-var='date'> 9</var>, <var data-var='time'>11:52</var> UTC Identified - The team has identified the root cause: hardware degradation affecting this model's hosting environment is causing backend instability. We are currently implementing a fix.Mar <var data-var='date'> 9</var>, <var data-var='time'>08:26</var> UTC Investigating - Our Model Hub Team is currently working on resolving errors related to an instance running the llama 405b model.

AI Model Hub: Increased Error Rate

Mar <var data-var='date'> 9</var>, <var data-var='time'>23:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'> 9</var>, <var data-var='time'>21:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Feb <var data-var='date'>28</var>, <var data-var='time'>15:33</var> UTC Scheduled - We will be performing a scheduled maintenance activity on the TXL network fabric. We do not expect any customer facing service degradation.

Scheduled Maintenance – BGP Route Advertisement Update (TXL)

Mar <var data-var='date'> 5</var>, <var data-var='time'>23:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'> 5</var>, <var data-var='time'>21:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'> 5</var>, <var data-var='time'>18:07</var> UTC Scheduled - To improve network stability in one cluster, we are rolling out a configuration update related to route advertisement mechanisms. No customer impact is expected.

Emergency Maintenance: TXL Network Configuration Update

Mar <var data-var='date'> 5</var>, <var data-var='time'>21:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'> 5</var>, <var data-var='time'>18:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Feb <var data-var='date'>28</var>, <var data-var='time'>15:43</var> UTC Scheduled - We are updating software packages in our network stack. We do not expect customer facing service degradation.

Scheduled Maintenance :Network Software Updates in FRA, TXL, FKB

Mar <var data-var='date'> 4</var>, <var data-var='time'>16:39</var> UTC Resolved - We are marking this incident as resolved because no further issues were found in the setup. A Root‑Cause Analysis (RCA) will be published once the team has completed its analysis.Mar <var data-var='date'> 4</var>, <var data-var='time'>12:22</var> UTC Monitoring - The Kubernetes Team has deployed a mitigation to the issue which involved a version rollback of a component of the K8s control plane. We are currently monitoring the service recovery.Mar <var data-var='date'> 4</var>, <var data-var='time'>10:23</var> UTC Identified - Our Container Registry team has identified an issue in the underlying Kubernetes cluster serving a subset of images. The team is currently working on applying a fix for the issue.Mar <var data-var='date'> 4</var>, <var data-var='time'>09:23</var> UTC Investigating - We are currently investigating an increased error count on the IONOS Container Registry. Customers might be unable to pull images currently.

IONOS Container Registry

Mar <var data-var='date'> 4</var>, <var data-var='time'>14:30</var> UTC Completed - We decided to postpone this FKB maintenance for some later date (to be defined).Mar <var data-var='date'> 3</var>, <var data-var='time'>17:07</var> UTC Scheduled - We would like to inform you that an exceptional maintenance of the network is due to be carried out in our location FKB. We need to exchange some faulty HW that is covered by redundancy. We do not expect any impact on the network and storage connectivity. The maintenance activity may cause latency for short time but no connectivity loss. All other services are expected to operate normally. Availability and accessibility of your virtual data center resources will remain unaffected.

Is IONOS Cloud down?

IONOS Cloud status is DEGRADED

Components with issues

Lastest published events

Cloud Panel Maintenance

Performance Degradation Compute FRA

Managed Kubernetes: Service Degradation for Control Planes in FRA

IONOS Cloud Services, Components and Regions

Recent IONOS Cloud events

Stability Improvements: Subnet Manager Migration in TXL (2/2)

Stability Improvements: Subnet Manager Migration in TXL (1/2)

Cloud Panel Maintenance

Performance Degradation Compute FRA

Managed Kubernetes: Service Degradation for Control Planes in FRA

Maintenance of the Backup service

Get your FREE account now,
5 monitors included!

Is IONOS Cloud down?

IONOS Cloud status is DEGRADED

Components with issues

Lastest published events

Cloud Panel Maintenance

Performance Degradation Compute FRA

Managed Kubernetes: Service Degradation for Control Planes in FRA

IONOS Cloud Services, Components and Regions

Recent IONOS Cloud events

Stability Improvements: Subnet Manager Migration in TXL (2/2)

Stability Improvements: Subnet Manager Migration in TXL (1/2)

Cloud Panel Maintenance

Performance Degradation Compute FRA

Managed Kubernetes: Service Degradation for Control Planes in FRA

Maintenance of the Backup service

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!