Jul <var data-var='date'>27</var>, <var data-var='time'>18:41</var> UTC Resolved - Unusual load on the proxy, caused some degraded performances on the device URL. Everything is now working properly. Jul <var data-var='date'>27</var>, <var data-var='time'>17:06</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>27</var>, <var data-var='time'>17:06</var> UTC Update - We are continuing to investigate this issue. Jul <var data-var='date'>27</var>, <var data-var='time'>16:49</var> UTC Investigating - We're currently seeing some Gateway error on public device url. Investigation ongoing.

Elevated Error on Public Device Url

Jul <var data-var='date'> 9</var>, <var data-var='time'>18:24</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 7</var>, <var data-var='time'>17:37</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'> 7</var>, <var data-var='time'>16:19</var> UTC Update - We are continuing to work on a fix for this issue. Jul <var data-var='date'> 7</var>, <var data-var='time'>16:18</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'> 2</var>, <var data-var='time'>22:42</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'> 2</var>, <var data-var='time'>20:42</var> UTC Update - We are continuing to work on a fix for this issue. Jul <var data-var='date'> 2</var>, <var data-var='time'>14:25</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'> 2</var>, <var data-var='time'>14:03</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently looking into the issue.

Elevated API Errors

Jul <var data-var='date'> 6</var>, <var data-var='time'>14:37</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 6</var>, <var data-var='time'>13:53</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'> 6</var>, <var data-var='time'>13:47</var> UTC Identified - We're experiencing an elevated level of API errors and are currently looking into the issue.

Jun <var data-var='date'>14</var>, <var data-var='time'>18:00</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>14</var>, <var data-var='time'>17:47</var> UTC Verifying - Verification is currently underway for the maintenance items. Jun <var data-var='date'>14</var>, <var data-var='time'>17:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'> 7</var>, <var data-var='time'>15:54</var> UTC Update - We will be undergoing scheduled maintenance during this time. Jun <var data-var='date'>14</var>, <var data-var='time'>17:00</var> UTC Scheduled - We will be undergoing scheduled maintenance during this time.

balenaCloud infrastructure maintenance

May <var data-var='date'> 5</var>, <var data-var='time'>21:03</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 5</var>, <var data-var='time'>18:43</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'> 5</var>, <var data-var='time'>18:19</var> UTC Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'> 5</var>, <var data-var='time'>18:12</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently looking into the issue.

Apr <var data-var='date'>21</var>, <var data-var='time'>16:30</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'> 8</var>, <var data-var='time'>19:51</var> UTC Update - We are continuing to monitor for any further issues. Apr <var data-var='date'> 8</var>, <var data-var='time'>19:51</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'>31</var>, <var data-var='time'>12:55</var> UTC Identified - We're experiencing an elevated level of errors in our application builder infrastructure and are currently looking into the issue.

Elevated GIT/Application Builder Errors

Apr <var data-var='date'>20</var>, <var data-var='time'>02:00</var> UTC Completed - The scheduled maintenance has been completed. Apr <var data-var='date'>20</var>, <var data-var='time'>00:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>20</var>, <var data-var='time'>00:36</var> UTC Scheduled - We will be undergoing scheduled maintenance during this time.

Mar <var data-var='date'>25</var>, <var data-var='time'>14:07</var> UTC Resolved - We experienced degraded API performance due to an internal configuration change that unintentionally increased system load, resulting in slower response times and reduced request capacity. Our team identified the issue and rolled back the change to restore stability. A root cause fix has been implemented and deployed aswell. The system has now fully recovered, and services are operating normally. Mar <var data-var='date'>24</var>, <var data-var='time'>13:21</var> UTC Update - We are continuing to monitor for any further issues. Mar <var data-var='date'>24</var>, <var data-var='time'>12:58</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'>24</var>, <var data-var='time'>12:43</var> UTC Identified - We're experiencing an elevated level of API errors and are currently looking into the issue.

Mar <var data-var='date'>25</var>, <var data-var='time'>13:18</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>23</var>, <var data-var='time'>18:55</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'>23</var>, <var data-var='time'>16:57</var> UTC Investigating - We are seeing several builds intermitently failing with 404 errors - No such image during builds and are investigating.

Builder Degraded performance

Mar <var data-var='date'> 3</var>, <var data-var='time'>20:17</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 3</var>, <var data-var='time'>19:45</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'> 3</var>, <var data-var='time'>17:44</var> UTC Investigating - We're experiencing an issue where the Dashboard may redirect to an unexpected page on initial load, which can prevent access to certain account and billing pages.

Elevated Dashboard Errors

Feb <var data-var='date'>27</var>, <var data-var='time'>03:39</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>27</var>, <var data-var='time'>03:02</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>27</var>, <var data-var='time'>02:41</var> UTC Identified - The issue has been identified and a fix is being implemented. Feb <var data-var='date'>26</var>, <var data-var='time'>21:13</var> UTC Update - We are continuing to investigate this issue. It only affects the direct SSH to device connection for example established via the balena-cli with balena device ssh <uuid>. Web Terminal via the dashboard and standard tunnelling is not affected. Feb <var data-var='date'>26</var>, <var data-var='time'>19:20</var> UTC Investigating - We're experiencing an elevated level of device SSH errors and are currently looking into the issue.

Elevated Device SSH Errors

Feb <var data-var='date'>24</var>, <var data-var='time'>20:54</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>24</var>, <var data-var='time'>18:41</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>24</var>, <var data-var='time'>18:29</var> UTC Identified - The issue has been identified and a fix is being implemented. Feb <var data-var='date'>24</var>, <var data-var='time'>17:29</var> UTC Update - We are continuing to investigate this issue. Feb <var data-var='date'>24</var>, <var data-var='time'>17:27</var> UTC Investigating - We're experiencing an elevated level of device SSH errors and are currently looking into the issue.

Feb <var data-var='date'>11</var>, <var data-var='time'>00:02</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>10</var>, <var data-var='time'>10:25</var> UTC Monitoring - We have identified the potential cause and have rolled back the changes. Feb <var data-var='date'>10</var>, <var data-var='time'>09:33</var> UTC Investigating - Some delta generation requests are encountering errors and failing. We are currently investigating this issue.

Elevated Delta Errors

Feb <var data-var='date'>10</var>, <var data-var='time'>07:56</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>10</var>, <var data-var='time'>07:17</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>10</var>, <var data-var='time'>06:39</var> UTC Identified - The issue has been identified and a fix is being implemented. Feb <var data-var='date'>10</var>, <var data-var='time'>04:05</var> UTC Investigating - We're experiencing an elevated level of errors in our Cloudlink infrastructure and are currently looking into the issue.

Elevated Cloudlink Errors

Feb <var data-var='date'> 6</var>, <var data-var='time'>13:53</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 5</var>, <var data-var='time'>17:56</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'> 4</var>, <var data-var='time'>20:43</var> UTC Investigating - We're experiencing an elevated level of errors in our application builder infrastructure and are currently looking into the issue.

Jan <var data-var='date'>28</var>, <var data-var='time'>22:06</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>28</var>, <var data-var='time'>20:08</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jan <var data-var='date'>28</var>, <var data-var='time'>19:45</var> UTC Identified - The issue has been identified and a fix is being implemented. Jan <var data-var='date'>28</var>, <var data-var='time'>18:54</var> UTC Investigating - We're experiencing an elevated level of API errors and are currently looking into the issue.

Jan <var data-var='date'>28</var>, <var data-var='time'>01:00</var> UTC Resolved - A configuration change causing multiple component restarts resulted in the API being unavailable for a few minutes.

Partial API Outage

Dec <var data-var='date'>24</var>, <var data-var='time'>09:38</var> UTC Resolved - Insufficient AWS compute capacity overloaded the remaining nodes. This high load caused readiness probes to fail, triggering API restarts that created a feedback loop of increasing pressure. Dec <var data-var='date'>23</var>, <var data-var='time'>19:03</var> UTC Monitoring - Scaling issues during service deployment caused by unavailability of nodes from underlying scaling provider. Dec <var data-var='date'>23</var>, <var data-var='time'>16:32</var> UTC Investigating - We are currently investigating an issue affecting the availability of balenaCloud services.

Degraded Performance

Dec <var data-var='date'> 5</var>, <var data-var='time'>09:37</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'> 5</var>, <var data-var='time'>09:16</var> UTC Monitoring - Our upstream provider has implemented some fixes. balenaCloud services are back online. We are still monitoring the situation. Dec <var data-var='date'> 5</var>, <var data-var='time'>09:05</var> UTC Identified - CloudFlare, our proxy provider, is having service issues. Connectivity to balenaCloud services are currently affected.

An upstream provider outage is affecting connectivity to balenaCloud services

Nov <var data-var='date'>24</var>, <var data-var='time'>03:00</var> UTC Completed - The scheduled maintenance has been completed. Nov <var data-var='date'>24</var>, <var data-var='time'>01:55</var> UTC Update - Due to some delays in infrastructure provisioning, we are extending maintenance for another hour. Nov <var data-var='date'>24</var>, <var data-var='time'>00:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>17</var>, <var data-var='time'>00:37</var> UTC Scheduled - We will be undergoing scheduled maintenance during this time.

Nov <var data-var='date'>12</var>, <var data-var='time'>19:58</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'>12</var>, <var data-var='time'>17:14</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Nov <var data-var='date'>12</var>, <var data-var='time'>15:01</var> UTC Identified - We have observed packet loss in the connections to our builder workers. We are working with our upstream provider to fix the issue. We have also provisioned a few replacement workers in an alternative location while we mitigate the problem. Nov <var data-var='date'>12</var>, <var data-var='time'>09:13</var> UTC Investigating - We are currently investigating this issue.

Builder partially degraded service

Oct <var data-var='date'>20</var>, <var data-var='time'>23:03</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>12:49</var> UTC Monitoring - The underlying issue of our upstream hosting provider has been mitigated. We are continuing to monitor the situation. Oct <var data-var='date'>20</var>, <var data-var='time'>08:45</var> UTC Identified - Our upstream hosting provider is currently having an outage. We have identified that downloads from the balena registry may be slow or disrupted. Builds may also fail if the images are being pulled from Dockerhub. We are monitoring the situation.

Oct <var data-var='date'> 1</var>, <var data-var='time'>13:30</var> UTC Resolved - An update to our kube-system infrastructure resulted in a disruptive pod rollout that left some devices disconnected from Cloudlink for up to 10 minutes. We apologize for the interruption and future updates to this component will be handled within planned maintenance windows.

Elevated Device URLs/VPN Errors

Sep <var data-var='date'>30</var>, <var data-var='time'>21:56</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'>30</var>, <var data-var='time'>21:34</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'>30</var>, <var data-var='time'>20:35</var> UTC Identified - The issue has been identified and a fix is being implemented. Sep <var data-var='date'>30</var>, <var data-var='time'>19:47</var> UTC Investigating - We're experiencing an elevated level of errors in our Device VPN Tunnel infrastructure and are currently looking into the issue.

Elevated Device VPN Tunnel Errors

Sep <var data-var='date'>25</var>, <var data-var='time'>08:57</var> UTC Resolved - Large files in build context caused increases of memory usage on builder-services. Higher memory allocations have been implemented resolving the buffer memory issues. Sep <var data-var='date'>23</var>, <var data-var='time'>10:27</var> UTC Update - We have detected that network latency between different regions is causing issues on build context processing for build contexts containing larger files. We will deploy new servers temporarily to mitigate the impact while we work on a permanent fix. Sep <var data-var='date'>16</var>, <var data-var='time'>17:26</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'>16</var>, <var data-var='time'>17:01</var> UTC Identified - The issue has been identified and a fix is being implemented. Sep <var data-var='date'>16</var>, <var data-var='time'>02:33</var> UTC Investigating - We are currently investigating this issue.

Is balena.io down?

balena.io status is UP

Lastest published events

Elevated Error on Public Device Url

Elevated API Errors

Elevated API Errors

balena.io Services, Components and Regions

Recent balena.io events

Elevated Error on Public Device Url

Elevated API Errors

Elevated API Errors

balenaCloud infrastructure maintenance

Elevated API Errors

Elevated GIT/Application Builder Errors