Aug <var data-var='date'>20</var>, <var data-var='time'>19:30</var> UTC Resolved - A BGP configuration error caused our Anycast DNS to route to some nodes without the proper DNS infrastructure. The issue was temporary and was resolved as soon as we removed that node from BGP.

Temporary DNS resolution failure

Aug <var data-var='date'>20</var>, <var data-var='time'>14:16</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>20</var>, <var data-var='time'>14:05</var> UTC Monitoring - A fix has been deployed and this error should no longer be occurring. We're monitoring to ensure full recovery. Aug <var data-var='date'>20</var>, <var data-var='time'>13:54</var> UTC Identified - We have identified an issue causing authentication errors for some operations from `flyctl`. These operations are failing with an error like: `This endpoint no longer accepts legacy OAuth tokens (starting with `fo1_`). Please use a macaroon token (starting with `fm2_`) instead. We have identified the issue and are rolling out a fix

Oauth/Macaroon Errors from flyctl

Aug <var data-var='date'>20</var>, <var data-var='time'>07:57</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>20</var>, <var data-var='time'>07:32</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>20</var>, <var data-var='time'>07:26</var> UTC Update - We are continuing to investigate this issue. Aug <var data-var='date'>20</var>, <var data-var='time'>07:25</var> UTC Investigating - We had an issue with the ord-0 Fly Kubernetes cluster, and many MPG clusters are failing to restart. Our MPG team is actively working on it.

MPG (v1) partially down in ORD

Aug <var data-var='date'>19</var>, <var data-var='time'>00:00</var> UTC Resolved - 6PN networking issues between some machines in YYZ during a rollout which was rolled back once we noticed errors. During this time some machines were unable to talk to internal resources like other DBs, other apps or MPG clusters.

6PN Networking issue in YYZ

Aug <var data-var='date'>17</var>, <var data-var='time'>15:41</var> UTC Resolved - The capacity issue in the ARN region has been resolved. Aug <var data-var='date'>17</var>, <var data-var='time'>13:19</var> UTC Investigating - New machines may fail to create in ARN because we lack capacity.

No capacity in ARN

Aug <var data-var='date'>14</var>, <var data-var='time'>20:33</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>14</var>, <var data-var='time'>20:08</var> UTC Monitoring - We have failed over the secrets database to a replica, and the Machines API appears healthy now. We are monitoring for any further issues. Aug <var data-var='date'>14</var>, <var data-var='time'>19:30</var> UTC Identified - We are working to recover our secrets service after a failed deployment. Apps continue to run, but it is not possible to create new apps or update secrets at this time.

Secrets service outage

Aug <var data-var='date'>13</var>, <var data-var='time'>19:41</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>13</var>, <var data-var='time'>18:37</var> UTC Update - We are continuing to monitor for any further issues. Aug <var data-var='date'>13</var>, <var data-var='time'>18:36</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>13</var>, <var data-var='time'>18:14</var> UTC Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>13</var>, <var data-var='time'>17:51</var> UTC Update - We are continuing to investigate this issue. Aug <var data-var='date'>13</var>, <var data-var='time'>17:45</var> UTC Investigating - We are currently investigating degraded ipv6 networking on a subset of hosts

IPv6 Networking Issues

Aug <var data-var='date'>13</var>, <var data-var='time'>16:00</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>13</var>, <var data-var='time'>13:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'> 7</var>, <var data-var='time'>21:11</var> UTC Scheduled - An upstream provider is performing network maintenance on a subset of hosts in ORD on 2026-08-13, from 13:00 UTC to 16:00 UTC. A loss of connectivity of up to 30 seconds is possible within the scheduled maintenance window.

Network maintenance in ORD

Aug <var data-var='date'> 9</var>, <var data-var='time'>07:10</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'> 9</var>, <var data-var='time'>06:42</var> UTC Monitoring - A fix has been implemented and we are monitoring the results Aug <var data-var='date'> 9</var>, <var data-var='time'>06:03</var> UTC Update - We’ve deployed an additional mitigation to further reduce Corrosion retry pressure and are seeing improvement; we’re continuing to monitor while remaining affected nodes catch up. Aug <var data-var='date'> 9</var>, <var data-var='time'>04:58</var> UTC Update - We’ve applied a mitigation to reduce the impact from Corrosion batch insertion retries and are continuing to monitor while affected nodes catch up. Aug <var data-var='date'> 9</var>, <var data-var='time'>03:50</var> UTC Identified - We have identified the issue as failed insertions in a subset of Corrosion batches. These failures trigger retries, which can cause timeouts for other batches. Aug <var data-var='date'> 9</var>, <var data-var='time'>02:50</var> UTC Investigating - We are currently investigating app-not-found errors returned by Machines API calls made shortly after creating new applications.

Increased app-not-found errors

Aug <var data-var='date'> 5</var>, <var data-var='time'>20:41</var> UTC Resolved - Etcd is stable. Services are back to normal. Aug <var data-var='date'> 5</var>, <var data-var='time'>20:26</var> UTC Identified - High CPU pressure on a shared etcd instance is causing lags on MPG creation in the IAD region

MPG IAD data plane degraded for new clusters

Aug <var data-var='date'> 5</var>, <var data-var='time'>03:22</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'> 4</var>, <var data-var='time'>20:59</var> UTC Monitoring - We tweaked hosts to allow for more machine allocation. We'll be monitoring the region over the next hours. Aug <var data-var='date'> 4</var>, <var data-var='time'>19:03</var> UTC Identified - New MPG clusters may fail to create in GRU because we lack capacity.

MPG creation is failing in GRU due to lack of capacity

Aug <var data-var='date'> 4</var>, <var data-var='time'>23:58</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'> 4</var>, <var data-var='time'>20:36</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'> 4</var>, <var data-var='time'>18:26</var> UTC Update - We have the size of TLS certificate issuance backlog under control, but are still seeing some remaining issues and are currently working to clean up the edge cases. Aug <var data-var='date'> 4</var>, <var data-var='time'>17:02</var> UTC Identified - We believe we have identified the issue and are releasing a fix. Aug <var data-var='date'> 4</var>, <var data-var='time'>14:53</var> UTC Investigating - We're currently investigating issues related to certificate issuance. Certificates may be delayed for new custom domains.

Certificate issuance delays

Aug <var data-var='date'> 3</var>, <var data-var='time'>18:50</var> UTC Resolved - A BGP misconfiguration while provisioning new edge capacity caused most traffic from Europe endpoints to be dropped, between 14:50 UTC and 15:02 UTC. The misconfiguration has been fixed and we are implementing safeguards against this kind of issue in the future.

Inbound connection failure to Fly Apps

Aug <var data-var='date'> 1</var>, <var data-var='time'>18:30</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'> 1</var>, <var data-var='time'>17:26</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'> 1</var>, <var data-var='time'>17:01</var> UTC Investigating - We are investigating an issue with the control plane for Managed Postgres v2 in the IAD region. Creating new v2 clusters in the IAD region may fail at this time. Existing clusters continue to run, but may experience lagging backups.

Managed Postgres v2 control plane issues in iad

Jul <var data-var='date'>31</var>, <var data-var='time'>20:11</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>31</var>, <var data-var='time'>19:02</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>31</var>, <var data-var='time'>16:44</var> UTC Investigating - Creating machines in CDG region may fail at this time with an "no capacity available in cdg" message. Existing apps continue to run.

Capacity issues in CDG

Jul <var data-var='date'>31</var>, <var data-var='time'>14:51</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>31</var>, <var data-var='time'>14:38</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>31</var>, <var data-var='time'>13:34</var> UTC Investigating - Our dashboard is failing to send outbound email. Emails such as new account verification or password reset may fail to send at this time.

Outbound email issues

Jul <var data-var='date'>29</var>, <var data-var='time'>13:26</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>29</var>, <var data-var='time'>12:41</var> UTC Investigating - We are investigating some database issues causing high latency on some API endpoints and dashboard operations. You may experience intermittent "503 service unavailable" errors at this time. Currently deployed apps continue to run.

Increased API latency

Jul <var data-var='date'>22</var>, <var data-var='time'>10:00</var> UTC Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>22</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>21</var>, <var data-var='time'>21:34</var> UTC Scheduled - An upstream network provider will be performing emergency maintenance in the DFW region, on 2026-07-22 between 08:00-10:00 UTC (03:00-05:00 local time). Apps will experience loss of connectivity for approximately 30 minutes within the scheduled maintenance window.

Emergency network maintenance in DFW

Jul <var data-var='date'>20</var>, <var data-var='time'>17:09</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>20</var>, <var data-var='time'>13:34</var> UTC Update - We are still working on fixing degraded Managed Postgres clusters. Jul <var data-var='date'>20</var>, <var data-var='time'>09:14</var> UTC Update - Some Managed Postgres v1 clusters are degraded. We are working on fixing them. Managed Postgres v2 is unaffected. Jul <var data-var='date'>20</var>, <var data-var='time'>08:02</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>20</var>, <var data-var='time'>07:52</var> UTC Update - We've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience. Jul <var data-var='date'>20</var>, <var data-var='time'>07:51</var> UTC Identified - We've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience. Jul <var data-var='date'>20</var>, <var data-var='time'>07:48</var> UTC Update - We are continuing to investigate this issue. Jul <var data-var='date'>20</var>, <var data-var='time'>07:10</var> UTC Investigating - Existing machines are unaffected. We are investigating the issue.

High number of 5XX on the Machines API and dashboard

Jul <var data-var='date'>19</var>, <var data-var='time'>04:52</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>19</var>, <var data-var='time'>03:48</var> UTC Identified - We have identified an upstream issue that is preventing egress IPv6 addresses in BOM from reaching parts of the internet, and we're currently working with an upstream provider to resolve this issue. Normal IPv6 addresses remain unaffected.

Egress IPv6 issues in BOM

Jul <var data-var='date'>18</var>, <var data-var='time'>03:00</var> UTC Completed - The scheduled maintenance has been completed. The maintenance ended up taking a little bit longer than the scheduled time - it fully wrapped up roughly one hour after the scheduled end time. Jul <var data-var='date'>18</var>, <var data-var='time'>01:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'> 8</var>, <var data-var='time'>18:29</var> UTC Scheduled - An upstream provider is performing network maintenance on a subset of hosts in FRA on 2026-07-18, from 01:00 UTC to 03:00 UTC. A loss of connectivity for up to 20 minutes is possible within the scheduled maintenance window.

Network maintenance in FRA

Jul <var data-var='date'>17</var>, <var data-var='time'>18:30</var> UTC Resolved - A bad deployment momentarily caused issues with Flycast connectivity, and, by extension, MPG, in our YYZ region. The deployment was immediately rolled back and connectivity was restored shortly after.

Brief Flycast / MPG interruption in YYZ

Jul <var data-var='date'>16</var>, <var data-var='time'>13:44</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>16</var>, <var data-var='time'>12:27</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>16</var>, <var data-var='time'>11:34</var> UTC Investigating - We are investigating increased connection latency and "connection reset" errors from our edge proxy. Apps continue to run, but requests may experience increased connection latency or fail at this time.

Edge proxy issues

Jul <var data-var='date'>16</var>, <var data-var='time'>12:17</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>16</var>, <var data-var='time'>11:54</var> UTC Identified - An issue with our Machines API is causing app creations to fail in some cases. We are working on a fix.

App creation failing

Jul <var data-var='date'>14</var>, <var data-var='time'>16:59</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>14</var>, <var data-var='time'>15:53</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Apps should be reachable at this point in time. Jul <var data-var='date'>14</var>, <var data-var='time'>14:51</var> UTC Identified - A subset of hosts in SJC are currently offline. Some apps may be unreachable at this time.

Is Fly.io down?

Fly.io status is UP

Lastest published events

Temporary DNS resolution failure

Oauth/Macaroon Errors from flyctl

MPG (v1) partially down in ORD

Fly.io Services, Components and Regions

Recent Fly.io events

Temporary DNS resolution failure

Oauth/Macaroon Errors from flyctl

MPG (v1) partially down in ORD

6PN Networking issue in YYZ

No capacity in ARN

Secrets service outage