Jun <var data-var='date'>12</var>, <var data-var='time'>10:29</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>12</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>12</var>, <var data-var='time'>06:22</var> UTC Scheduled - There could be a short downtime when switching the database instances in the cluster to perform the upgrade.

Database upgrade

May <var data-var='date'>29</var>, <var data-var='time'>06:00</var> UTC Resolved - Following a scheduled TLS certificate renewal, a subset of Netdata agents failed to reconnect to Netdata Cloud. The issue stems from outdated CA (Certificate Authority) trust stores on certain legacy Linux distributions, like: • CentOS 7 / RHEL 7 • Amazon Linux 2 • Debian 9 (Jessie) • Ubuntu 16.04 LTS • Oracle Linux 7 A quick way to test for this: curl -v https://app.netdata.cloud If the output shows an error indicating the certificate has expired, the system needs an updated CA bundle.

Agent connectivity issue on legacy systems

May <var data-var='date'> 8</var>, <var data-var='time'>06:00</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 8</var>, <var data-var='time'>01:13</var> UTC Monitoring - We are currently tracking an issue with our upstream cloud provider that is affecting a subset of our Kubernetes nodes. As a result, some customers may receive false-positive alerts regarding node reachability. Please be assured that your underlying services and workloads remain operational, and these alerts can be safely ignored at this time. We are closely monitoring the situation with our provider.

False-Positive Node Reachability Alerts

May <var data-var='date'> 8</var>, <var data-var='time'>00:32</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 8</var>, <var data-var='time'>00:12</var> UTC Update - We are continuing to monitor for any further issues. May <var data-var='date'> 8</var>, <var data-var='time'>00:11</var> UTC Monitoring - We are currently tracking an issue with our upstream cloud provider that is affecting a subset of our Kubernetes nodes. As a result, some customers may receive false-positive alerts regarding node reachability. Please be assured that your underlying services and workloads remain operational, and these alerts can be safely ignored at this time. We are closely monitoring the situation with our provider.

Apr <var data-var='date'>15</var>, <var data-var='time'>06:38</var> UTC Resolved - At ~02:20 UTC, a Kubernetes node running part of our ingress controller failed and was auto-replaced. Some clients experienced brief agent disconnections and false-positive node-unavailability alerts during the failover. Service has fully recovered after few minutes. We apologize for the inconvenience.

Kubernetes node failure.

Mar <var data-var='date'>24</var>, <var data-var='time'>10:48</var> UTC Completed - The scheduled maintenance has been completed. Mar <var data-var='date'>24</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>17</var>, <var data-var='time'>12:15</var> UTC Scheduled - A maintenance window is required to update the configuration of our MQTT broker. All clients will be temporarily disconnected during this process. The interruption is expected to be brief, after which normal connectivity will resume. Thank you for your understanding.

Scheduled maintenance on MQTT Broker

Mar <var data-var='date'> 8</var>, <var data-var='time'>15:30</var> UTC Resolved - Yesterday, a brief infrastructure incident occurred due to certain processes on our load balancer failing (exactly at 16:34 UTC). This caused more than half of our globally connected nodes to reconnect, which in turn triggered false positive alerts notifying clients of node downtime. We apologize for the confusion and are taking steps to ensure this does not happen again.

Unexpected Loadbalancer restart

Jan <var data-var='date'>22</var>, <var data-var='time'>15:20</var> UTC Resolved - In addition to the fix for the repository configuration packages, a fix has been deployed to the repositories themselves that will enable affected systems to update successfully to the latest version of the repository configuration packages, and as such no further issues are expected. Jan <var data-var='date'>22</var>, <var data-var='time'>14:06</var> UTC Monitoring - The primary fix has been deployed and appears to be working correctly. We are currently working on a followup fix to allow affected systems to update properly without requiring user intervention. Until that additional fix is deployed, affected systems can be manually updated to fully resolve the issue locally by fetching the latest version of the repository configuration package manually and installing it on the system: - For most RHEL 8 equivalent systems using our stable repositories: https://repository.netdata.cloud/repos/repoconfig/el/8/x86_64/netdata-repo-5-4.noarch.rpm - For most RHEL 8 equivalent systems using our nightly repositories: https://repository.netdata.cloud/repos/repoconfig/el/8/x86_64/netdata-repo-edge-5-4.noarch.rpm - For Oracle Linux 8 systems using our stable repositories: https://repository.netdata.cloud/repos/repoconfig/ol/8/x86_64/netdata-repo-5-4.noarch.rpm - For Oracle Linux 8 systems using our nightly repositories: https://repository.netdata.cloud/repos/repoconfig/ol/8/x86_64/netdata-repo-edge-5-4.noarch.rpm Jan <var data-var='date'>22</var>, <var data-var='time'>12:48</var> UTC Update - We are continuing to work on a fix for this issue. Jan <var data-var='date'>22</var>, <var data-var='time'>12:47</var> UTC Identified - We have identified an issue with the latest version of the repository configuration packages for our RHEL 8.x repositories that causes systems which update to that version of the repository configuration package to become unable to access our repositories. This affects all systems that use our repository configuration packages to access our RHEL 8.x repositories, including but not limited to RHEL 8.x, Alma Linux 8.x, Rocky Linux 8.x, and Oracle Linux 8.x. Affected systems will report a failure to download repository metadata for Netdata repositories. A fix is currently being tested and is expected to be deployed soon.

404 errors with Netdata RHEL 8.x repositories with latest repository configuration packages.

Dec <var data-var='date'> 2</var>, <var data-var='time'>10:10</var> UTC Completed - The scheduled maintenance has been completed. Dec <var data-var='date'> 2</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>24</var>, <var data-var='time'>14:50</var> UTC Scheduled - We’re upgrading the MQTT broker and its backend service. During this maintenance window, all agent connections will be reset, causing a brief service interruption. We apologize for any inconvenience.

MQTT Broker upgrade

Nov <var data-var='date'>18</var>, <var data-var='time'>19:22</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'>18</var>, <var data-var='time'>12:23</var> UTC Monitoring - Connectivity is recovering, we'll keep this open until we see traffic at previous levels. See https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 for details on the Cloudflare outage itself. Nov <var data-var='date'>18</var>, <var data-var='time'>12:11</var> UTC Investigating - As Cloudflare is experiencing network issues, Netdata Cloud, as well as our package repositories are currently unavailable.

Services Unavailable

Sep <var data-var='date'>30</var>, <var data-var='time'>18:29</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'>30</var>, <var data-var='time'>05:04</var> UTC Monitoring - We have released Netdata Agent v2.7.1 and recommend all Windows users to upgrade. Release notes: https://github.com/netdata/netdata/releases/tag/v2.7.1 Sep <var data-var='date'>29</var>, <var data-var='time'>11:36</var> UTC Identified - We've identified an issue with the latest Agent release on Windows that causes the go.d plugin to not be started. The impact is that for Windows nodes, none of the metrics from the go.d plugin are being collected and corresponding alerts don't fire. We will be releasing a patch release to address this as soon as possible today. As a workaround the plugin binary in Program Files can be renamed to add the missing .exe extension.

Agent v2.7.0 on Windows doesn't start go.d plugin

Aug <var data-var='date'>28</var>, <var data-var='time'>10:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>28</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>26</var>, <var data-var='time'>13:29</var> UTC Scheduled - We're changing the configuration of the load balancer. Not expecting any downtimes.

Changes in our infrastructure

Aug <var data-var='date'>21</var>, <var data-var='time'>19:58</var> UTC Resolved - All repositories are functioning normally again. Aug <var data-var='date'>21</var>, <var data-var='time'>19:42</var> UTC Monitoring - A fix has been implemented and we expect the repositories to be functioning normally within the next 30 minutes. Aug <var data-var='date'>21</var>, <var data-var='time'>19:39</var> UTC Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>21</var>, <var data-var='time'>19:35</var> UTC Investigating - We are currently investigating issues with availability of our nightly DEB and RPM packages which are preventing updates and installs from working correctly.

Package availability issues with native DEB/RPM packages.

Aug <var data-var='date'>20</var>, <var data-var='time'>13:09</var> UTC Resolved - This incident has been resolved. We recommend that Agents are upgraded to v2.6.2 on the stable channel, or the latest nightly build at the earliest convenience. Aug <var data-var='date'>13</var>, <var data-var='time'>15:30</var> UTC Monitoring - We've implement a change for this issue which appeared in our latest nightly, and resolved the problem. We are now issuing a new stable release v2.6.2 to include that fix, and will resolve this issue when all packages are available. Aug <var data-var='date'>12</var>, <var data-var='time'>09:28</var> UTC Identified - We have identified the cause of this issue and have been creating a fix which will appear in the upcoming nightly build. When we are confident that there are no negative effects on that build, we will do a new patch release on the stable channel the next day. What happened is that we were not correctly clearing MQTT messages that had been acknowledged by the broker, causing newer messages to not be correctly recognized as acknowledged in certain conditions. Aug <var data-var='date'>11</var>, <var data-var='time'>14:24</var> UTC Update - We will be adding additional debugging code in both the Agent (next nightly) and Cloud Backend to track what is causing these spurious latencies. Aug <var data-var='date'>11</var>, <var data-var='time'>12:27</var> UTC Investigating - We are investigating an issue with increased data request latencies that can result in slow or completely empty charts for some customers. The affected Netdata Agent versions are v2.6.0 and later, for all stable and nightly builds.

Increased data request latencies

Aug <var data-var='date'>14</var>, <var data-var='time'>15:12</var> UTC Resolved - We have confirmed that the issue that caused the incomplete package uploads has been fully fixed, and all packages that should have been uploaded since the issue first occurred have been fully uploaded and are available in the repositories. Aug <var data-var='date'>14</var>, <var data-var='time'>15:03</var> UTC Monitoring - Due to issues in our package upload process, the v2.6.2 stable release and the nightly builds for 2025-08-13 and 2025-08-14 were not fully uploaded to our official DEB/RPM repositories. We have already identified the root cause and implemented a fix, and are currently monitoring the processing of the package upload backlog to ensure that all of the packages are properly available.

Issues with package availability in official DEB/RPM package repositories

Jul <var data-var='date'>28</var>, <var data-var='time'>10:44</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>25</var>, <var data-var='time'>12:27</var> UTC Monitoring - We identified monitoring issues affecting customers using Netdata Agent version 2.6.0 on Windows servers. These issues were caused by upstream changes in Windows libraries that impacted agent functionality. Resolution Fixed Version Available: Netdata Agent v2.6.1 Release Date: July 25, 2025 (16:00 UTC) Preview Available: Nightly release v2.6.0-2 Required Action Immediate upgrade recommended - All users running Netdata Agent v2.6.0 on Windows servers should upgrade to v2.6.1 as soon as possible to restore full monitoring capabilities. We appreciate your patience as we continue working on additional improvements related to the upstream Windows library changes.

Windows Agent Monitoring Issues

Jun <var data-var='date'> 9</var>, <var data-var='time'>11:00</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'> 9</var>, <var data-var='time'>09:46</var> UTC Verifying - Verification is currently underway for the maintenance items. Jun <var data-var='date'> 9</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'> 9</var>, <var data-var='time'>11:54</var> UTC Update - Ensure your firewalls are configured correctly to target our hostnames rather than IP addresses. You can find a full list of endpoints here: https://learn.netdata.cloud/docs/netdata-cloud/agent-cloud-link-aclk Apr <var data-var='date'> 9</var>, <var data-var='time'>11:40</var> UTC Scheduled - We’re upgrading our infrastructure, which will unfortunately end the Agent-Cloud connection for older agent versions (1.42.0 and earlier). Please update your agents to the newest version before the cutoff date.

Important Notice: Changes Impacting Older Agent Versions (1.42.0 and earlier)

Jun <var data-var='date'> 6</var>, <var data-var='time'>10:30</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'> 6</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'> 5</var>, <var data-var='time'>07:39</var> UTC Scheduled - We will be upgrading our database instance. We expect a small downtime only when switching between HA instances.

Jun <var data-var='date'> 4</var>, <var data-var='time'>13:49</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'> 4</var>, <var data-var='time'>13:42</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 4</var>, <var data-var='time'>12:59</var> UTC Identified - We had a major problem with our MQTT broker. It is currently up and running and agents are reconnecting to the cloud.

MQTT broker failure

Apr <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> UTC Completed - The scheduled maintenance has been completed. Apr <var data-var='date'> 8</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'> 7</var>, <var data-var='time'>11:17</var> UTC Scheduled - As announced by e-mail to affected customers, we are implementing a restriction to actively deny older Agents from connecting to Netdata Cloud. After the restriction is put in place, all Agents with versions older than 1.40.0 will be automatically disconnected. These Agent versions have been out of support for over 12 months. We recommend upgrading to the most recent stable release (v2.3.2) before the start of the maintenance window, to ensure uninterrupted service. Note that we only <a href="https://learn.netdata.cloud/docs/netdata-agent/versions-&-platforms#binary-distribution-packages">provide native packages</a> for platforms that are still officially supported by their vendors. If you need to upgrade Netdata Agent on a platform for which we no longer publish newer native packages, you can switch your install to use a static build. While this is an unsupported procedure, you can find instructions <a href="https://github.com/netdata/netdata/pull/20075">here</a>.

Implement restriction to prevent Agents versions older than 1.40.0 from connecting to Netdata Cloud

Apr <var data-var='date'> 2</var>, <var data-var='time'>10:00</var> UTC Completed - The scheduled maintenance has been completed. Apr <var data-var='date'> 2</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>27</var>, <var data-var='time'>09:59</var> UTC Scheduled - During this maintenance window, we will: - Switch to a new DDoS provider (the main driver for the extended duration). We have tested it in non-production environments and expect no downtime, but IP addresses will change. Please update firewalls to use DNS names. We will trigger Agent-Cloud link reconnection but it should be transparent for end users. - Upgrade Elasticsearch, which forms the backbone of our Event Feed. We do expect Events tab to not work properly.

DDoS provider change, Events DB upgrade

Apr <var data-var='date'> 2</var>, <var data-var='time'>09:25</var> UTC Resolved - Affected Agents can cause the creation of multiple duplicate nodes in Netdata Cloud. All but the last one will appear as offline, and the last one will be as if it was created from scratch with no data. Unfortunately, the previously stored metrics for the affected nodes can not be recovered. The duplicate offline nodes can be safely deleted from Space Settings -> Nodes. Note that you may have to add the newest copy of these nodes to the appropriate rooms. The fixed nightly static build is v2.3.0-102. Apr <var data-var='date'> 1</var>, <var data-var='time'>09:00</var> UTC Investigating - We have found that a recent change in the nightly static builds of Netdata Agent causes meta data on the Agent to be overwritten. Specifically the sqlite3 database that keeps meta data on which timeseries stored in dbengine correspond with which metrics, and the information on the Agent's "machine GUID" are overwritten with the same copy in the build package. Not affected are: - All stable releases - Native packages (.deb and .rpm) Affected are all nightly static builds with the following version numbers: - 2.3.0-50-nightly - 2.3.0-60-nightly - 2.3.0-72-nightly - 2.3.0-78-nightly - 2.3.0-87-nightly The initial impact is that all affected Agent installs, even though they still have the timeseries data stored on disk, have lost all meta data associated with it, so these timeseries become inaccessable. This is unrecoverable. Additionally, the main form of identification is overwritten, too. We are assessing what the impact is for users of Netdata Cloud, and will update this incident with more information when the investigation is completed. The bug itself has been fixed and merged. We will issue a new nightly build shortly.

Nightly static builds overwrite node and metric data upon install

Mar <var data-var='date'>17</var>, <var data-var='time'>15:50</var> UTC Completed - All repositories appear to be working correctly with the new keys. Mar <var data-var='date'>17</var>, <var data-var='time'>15:24</var> UTC Verifying - Deployment of the new repository configuration packages using the updated signing keys has finished. We are verifying that the deployed configuration works correctly. Mar <var data-var='date'>17</var>, <var data-var='time'>15:11</var> UTC Update - The new signing keys have been deployed on our infrastructure, and updated repository configuration packages are being built and deployed. Mar <var data-var='date'>17</var>, <var data-var='time'>14:45</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>17</var>, <var data-var='time'>14:36</var> UTC Scheduled - We are rotating our DEB/RPM package and metadata signing key as there is a possibility the private key has been compromised. Repository metadata updates and package installs from our official DEB/RPM repositories are expected to fail due to missing signing keys until this process is completed. Users of our RPM package repositories will need to accept the new signing keys the next time they run updates after the maintenance window is over. Users of our DEB package repositories will need to update the netdata-repo or netdata-repo-edge package after the maintenance window is over to ensure they are using the new signing keys.

DEB/RPM signing key rotation.

Feb <var data-var='date'>25</var>, <var data-var='time'>15:25</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>25</var>, <var data-var='time'>13:29</var> UTC Update - We are continuing to monitor for any further issues. Feb <var data-var='date'>25</var>, <var data-var='time'>13:14</var> UTC Monitoring - The builds are completed, so we are watching out for any remaining related issues. Feb <var data-var='date'>25</var>, <var data-var='time'>11:04</var> UTC Identified - We have identified the issue, committed a fix, and initiated new nightly builds for all platforms. This will take several hours. In the mean time, please restart Netdata to work around the issue. Feb <var data-var='date'>25</var>, <var data-var='time'>09:41</var> UTC Investigating - We are investigating an issue with today's nightly (v2.2.0-245), causing alerting to not work ("health") and external plugins, including go.d, to not connect properly. This may be resolved by restarting the Agent. Stable versions of the Agent are not affected.

Lastest Agent nightly build (v2.2.0-245) broken at first start

Jan <var data-var='date'>30</var>, <var data-var='time'>10:49</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>30</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>29</var>, <var data-var='time'>11:37</var> UTC Scheduled - After additional tweaks we need to restart two databases. We only expect higher latency on Netdata Cloud.

Is Netdata down?

Netdata status is UP

Lastest published events

Database upgrade

Agent connectivity issue on legacy systems

False-Positive Node Reachability Alerts

Netdata Services, Components and Regions

Recent Netdata events

Database upgrade

Agent connectivity issue on legacy systems

False-Positive Node Reachability Alerts

False-Positive Node Reachability Alerts

Kubernetes node failure.

Scheduled maintenance on MQTT Broker