Apr <var data-var='date'> 8</var>, <var data-var='time'>12:23</var> UTC Resolved - This incident has been resolved.Apr <var data-var='date'> 8</var>, <var data-var='time'>11:02</var> UTC Update - We are continuing to monitor for any further issues.Apr <var data-var='date'> 8</var>, <var data-var='time'>11:00</var> UTC Monitoring - We have implemented a fix. We're monitoring the cluster for further issues.Apr <var data-var='date'> 8</var>, <var data-var='time'>08:34</var> UTC Investigating - We are currently investigating an issue with our metrics cluster.

Metrics currently experiencing issues

Apr <var data-var='date'> 7</var>, <var data-var='time'>18:17</var> UTC Resolved - This incident has been resolved.Apr <var data-var='date'> 7</var>, <var data-var='time'>15:39</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Apr <var data-var='date'> 7</var>, <var data-var='time'>15:17</var> UTC Identified - We have restored GraphQL and dashboard availability, but some actions (e.g. app state updates) may still be delayed.Apr <var data-var='date'> 7</var>, <var data-var='time'>15:08</var> UTC Investigating - We are investigating issues with our GraphQL API and web dashboard

GraphQL API / Dashboard Issues

Apr <var data-var='date'> 2</var>, <var data-var='time'>15:30</var> UTC Completed - The scheduled maintenance has been completed.Apr <var data-var='date'> 2</var>, <var data-var='time'>14:36</var> UTC Update - Maintenance has been extended by one hour. The new maintenance window is 12:30-15:30 UTC (5:30am-8:30am local time). We will continue to provide updates as necessary.Apr <var data-var='date'> 2</var>, <var data-var='time'>12:30</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Mar <var data-var='date'>18</var>, <var data-var='time'>20:56</var> UTC Scheduled - We will be performing scheduled network maintenance in the SJC (San Jose, California) region between 12:30-14:30 UTC (5:30am-7:30am local time). During this period a brief outage of ~5m is expected for each machine.

SJC Network Maintenance

Mar <var data-var='date'>29</var>, <var data-var='time'>16:01</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>29</var>, <var data-var='time'>15:35</var> UTC Update - We've freed up additional room in the SIN and AMS regions and are monitoring capacity.Mar <var data-var='date'>29</var>, <var data-var='time'>15:33</var> UTC Monitoring - We've freed up additional room in the SIN and AMS regions and are monitoring capacity.Mar <var data-var='date'>29</var>, <var data-var='time'>15:19</var> UTC Update - We are currently investigating capacity issues in SIN and AMS regions that are affecting: - Machine Create and Start events - Deployments, due to affected, degraded Remote Builders - Sprite startup from cold stateMar <var data-var='date'>29</var>, <var data-var='time'>15:13</var> UTC Update - This may also affect: - Remote builders in AMS and SIN regions, which could currently be experiencing degraded performance or failures. - Sprites starting from a cold state, which may experience failures in startingMar <var data-var='date'>29</var>, <var data-var='time'>15:00</var> UTC Identified - We are currently investigating elevated errors when creating and starting machines in the SIN and AMS regions. Choosing other regions to create or deploy may help in the meantime

Low Capacity in SIN and AMS regions

Mar <var data-var='date'>27</var>, <var data-var='time'>21:51</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>27</var>, <var data-var='time'>21:09</var> UTC Monitoring - With the additional capacity we've brought online, machine start failure rates in IAD have now recovered. We'll continue to monitor IAD capacity.Mar <var data-var='date'>27</var>, <var data-var='time'>19:21</var> UTC Identified - We've brought some additional capacity online in IAD and are seeing improvements, and we're continuing to work on adding more and freeing up additional room.Mar <var data-var='date'>27</var>, <var data-var='time'>18:47</var> UTC Update - We're continuing to evaluate our options for increasing short-term capacity in the IAD region.Mar <var data-var='date'>27</var>, <var data-var='time'>18:08</var> UTC Investigating - We're currently investigating capacity issues in IAD that is preventing machine starts (machine creates are currently unaffected). This may result in deploys failing to complete (even for apps outside of the IAD region). As a workaround, using legacy Fly builders explicitly located in another region (i.e., `FLY_REMOTE_BUILDER_REGION=lhr fly deploy --depot=false --recreate-builder`) may help in the meantime.

Low capacity in IAD

Mar <var data-var='date'>26</var>, <var data-var='time'>17:54</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>26</var>, <var data-var='time'>17:28</var> UTC Monitoring - We've implemented a fix and have seen error rates for machine creates in ORD drop off. We're continuing to monitor the results.Mar <var data-var='date'>26</var>, <var data-var='time'>16:50</var> UTC Identified - We've identified the cause of this increased failure rate and a fix is in progress. We are seeing most creates in ORD succeed at this time, though failure rate is still above baseline.Mar <var data-var='date'>26</var>, <var data-var='time'>16:08</var> UTC Update - We are continuing to investigate this issue. We are seeing 408 errors decreasing in ORD, though still above baseline.Mar <var data-var='date'>26</var>, <var data-var='time'>15:21</var> UTC Investigating - We are currently investigating elevated errors creating machines in the ORD (Chicago, Illinois) region. Users may see `failed to launch VM: request returned non-2xx status: 408` errors when creating, updating, or scaling machines in ORD. Existing, already running machines in the ORD region continue to run as normal.

Machine Creates Failing in ORD Region

Mar <var data-var='date'>26</var>, <var data-var='time'>14:19</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>26</var>, <var data-var='time'>13:16</var> UTC Identified - Some Managed Postgres clusters in FRA region are still unreachable, we are investigating this issue.Mar <var data-var='date'>26</var>, <var data-var='time'>13:14</var> UTC Monitoring - Apps and Managed Postgres clusters in FRA region should be back online at this time. We are monitoring for any further issues.Mar <var data-var='date'>26</var>, <var data-var='time'>12:37</var> UTC Investigating - We are investigating network issues in FRA region. Apps and/or Managed Postgres clusters in the region may be inaccessible at this time.

Network issues in FRA region

Mar <var data-var='date'>23</var>, <var data-var='time'>16:27</var> UTC Resolved - This incident is resolved, Grafana logs are now working properly.Mar <var data-var='date'>23</var>, <var data-var='time'>15:55</var> UTC Monitoring - We've deployed a fix and are monitoring the results. Logs are now be visible on Grafana.Mar <var data-var='date'>23</var>, <var data-var='time'>15:41</var> UTC Identified - Using the Logs panel in Grafana at https://fly-metrics.net/ will show a 502 error from the backend and won't show any logs. You can use `fly logs` or the live log viewer directly on https://fly.io/dashboard to view streaming logs for the time being.Mar <var data-var='date'>23</var>, <var data-var='time'>15:18</var> UTC Investigating - Using the Logs panel in Grafana at https://fly-metrics.net/ will show a 502 error from the backend and won't show any logs. You can use `fly logs` or the live log viewer directly on https://fly.io/dashboard to view streaming logs for the time being.

Backend errors when trying to use Grafana to view logs

Mar <var data-var='date'>23</var>, <var data-var='time'>13:19</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>21</var>, <var data-var='time'>08:26</var> UTC Update - Machine start success rates in DFW have improved but we are continuing to monitor and make further adjustments. We will provide updates as the situation progresses.Mar <var data-var='date'>20</var>, <var data-var='time'>12:45</var> UTC Update - In addition to freeing up existing capacity, the team has provisioned new capacity in DFW and we are monitoring the results.Mar <var data-var='date'>20</var>, <var data-var='time'>08:08</var> UTC Monitoring - We freed up some capacity on our workers to allow for successful Machine starts.Mar <var data-var='date'>20</var>, <var data-var='time'>07:26</var> UTC Investigating - The Machines start failure rate is elevated in DFW.

Machines failing to start in DFW

Mar <var data-var='date'>19</var>, <var data-var='time'>10:37</var> UTC Resolved - This incident has been resolved. We're unable to recover the lost metrics from that one hour.Mar <var data-var='date'>19</var>, <var data-var='time'>07:12</var> UTC Monitoring - We have implemented a fix. There has been approximately 1h of lost metrics from 06:07UTC. We're monitoring the cluster for further issuesMar <var data-var='date'>19</var>, <var data-var='time'>06:28</var> UTC Investigating - We are currently investigating an issue with our metrics cluster.

Mar <var data-var='date'>18</var>, <var data-var='time'>18:53</var> UTC Resolved - This incident has been resolved. Machine creates in DFW continue to work normally.Mar <var data-var='date'>18</var>, <var data-var='time'>12:40</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'>18</var>, <var data-var='time'>11:44</var> UTC Identified - The team is currently rolling out additional capacity in DFW which should help ease Machine start failures across the region.Mar <var data-var='date'>18</var>, <var data-var='time'>09:58</var> UTC Investigating - We are investigating reports of machines failing to start in the DFW (Dallas) region with "insufficient memory" errors. This may cause deployment failures for applications running in DFW. Our team is actively working to restore full capacity in the region. If you are affected, deploying to an alternate region may serve as a temporary workaround. We will provide updates as the situation progresses.

Mar <var data-var='date'>18</var>, <var data-var='time'>17:02</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>18</var>, <var data-var='time'>16:31</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'>18</var>, <var data-var='time'>16:12</var> UTC Investigating - We are investigating intermittent network issues in SJC region impacting outbound public IPv6 access from Machines. Connecting to IPv6 internet resources from apps hosted in SJC region may be slow or fail at this time. IPv4 access, as well as 6PN private networking, are unaffected.

IPv6 networking issues in SJC region

Mar <var data-var='date'>18</var>, <var data-var='time'>14:18</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>18</var>, <var data-var='time'>14:07</var> UTC Monitoring - Between 13:55 and 14:03 UTC machines and MPG clusters hosted in the SJC region saw elevated connection errors. Users may have seen errors connecting to or from most machines in the region, as well as with deployments or updates to machines in the region. Networking has returned to normal in the region, and we are continuing to monitor closely to ensure stable recovery.

Connection Issues in SJC

Mar <var data-var='date'>18</var>, <var data-var='time'>14:18</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>18</var>, <var data-var='time'>14:17</var> UTC Monitoring - A fix has been implemented and we are seeing `ssh console` commands succeed as normal.Mar <var data-var='date'>18</var>, <var data-var='time'>14:12</var> UTC Identified - We have identified an issue causing new `fly ssh console` connections to fail with 500 errors. A fix is in progress.

Fly ssh console command failing

Mar <var data-var='date'>14</var>, <var data-var='time'>14:05</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>14</var>, <var data-var='time'>04:20</var> UTC Monitoring - Organizations with names prefixed with numerical digits may experience 401 errors. Affected operations include actions such as Sprite creation, listing, etc... A fix has been implemented since 2026-03-14 12:30 UTC and we are monitoring the results!

Sprites Operations: 401 errors for certain organizations

Mar <var data-var='date'>11</var>, <var data-var='time'>11:37</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>11</var>, <var data-var='time'>11:03</var> UTC Update - While the secret storage service was in a read-only state, app creation requests queued up, due to the retry logic and insufficient request concurrency limits in our GraphQL API. This prevented our GraphQL API from serving any other requests. We have scaled up the GraphQL API and are continuing to monitor the situation.Mar <var data-var='date'>11</var>, <var data-var='time'>10:14</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'>11</var>, <var data-var='time'>09:19</var> UTC Identified - An ongoing data migration in our secret storage service is causing degraded Machines API functionality.

Setting secrets and creating apps is degraded

Mar <var data-var='date'> 7</var>, <var data-var='time'>15:56</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'> 7</var>, <var data-var='time'>15:10</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'> 7</var>, <var data-var='time'>14:42</var> UTC Investigating - We are investigating a private networking failure between SYD and other regions. Apps continue to run, and private networking within SYD is unaffected.

Private networking issues in SYD region

Mar <var data-var='date'> 5</var>, <var data-var='time'>19:50</var> UTC Resolved - This incident has been resolved. Due to a BGP issue, we saw some North American traffic routed to edges in Singapore (sin). Users in North America would have seen additional request latency during this period.Mar <var data-var='date'> 5</var>, <var data-var='time'>19:38</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'> 5</var>, <var data-var='time'>19:24</var> UTC Investigating - We're aware of routing issues affecting some customers in North America regions, and we're actively investigating.

Routing issues in NA regions

Mar <var data-var='date'> 4</var>, <var data-var='time'>09:00</var> UTC Completed - The scheduled maintenance has been completed.Mar <var data-var='date'> 4</var>, <var data-var='time'>03:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Feb <var data-var='date'>20</var>, <var data-var='time'>19:32</var> UTC Scheduled - An upstream provider is performing network maintenance in GRU on 2026-03-04, from 03:00 UTC (00:00am local time) to 09:00 UTC (6:00am local time). A loss of connectivity for up to 30 minutes is expected within the scheduled maintenance window.

Network Maintenance in GRU

Mar <var data-var='date'> 3</var>, <var data-var='time'>21:15</var> UTC Resolved - This incident was caused by a failed Redis node that powers our GraphQL API. We were able to recreate the Redis node and restore service. We are still investigating the root cause of the failure. In the mean time, all API endpoints now appear to be stable and errors have dropped to baseline level.Mar <var data-var='date'> 3</var>, <var data-var='time'>20:36</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'> 3</var>, <var data-var='time'>20:18</var> UTC Investigating - We're investigating elevated GraphQL errors that affect some API endpoints.

Elevated GraphQL API errors

Mar <var data-var='date'> 3</var>, <var data-var='time'>12:10</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'> 3</var>, <var data-var='time'>10:50</var> UTC Investigating - We are currently investigating this issue. The page currently displays: "We’re having trouble loading the cost breakdown."

Cost Explorer fails to load

Mar <var data-var='date'> 3</var>, <var data-var='time'>00:54</var> UTC Resolved - Between 19:54 and 20:06 UTC, our Vault cluster serving app certificates was unavailable. This caused various API requests to fail, mainly operations on certificates but also app creates and IP assignments. As the failure mode was Vault requests hanging rather than failing immediately, TLS requests through fly-proxy for domains where the certificate was not cached on the local node remained open for a long time while proxy attempted to fetch the certificate; this caused some connections to fail as too many connection slots were taken up by requests waiting on Vault. The root cause of this incident was a partially completed update to the Vault cluster. We will be implementing safeguards in the proxy for this failure mode, as well as improving certificate storage longer-term.

Certificates issues affecting API and proxy

Mar <var data-var='date'> 2</var>, <var data-var='time'>22:49</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'> 2</var>, <var data-var='time'>20:35</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'> 2</var>, <var data-var='time'>18:21</var> UTC Identified - The issue has been identified and a fix is being implemented.Mar <var data-var='date'> 2</var>, <var data-var='time'>17:42</var> UTC Investigating - We are currently investigating this issue.

Machines failing to boot in EWR

Mar <var data-var='date'> 2</var>, <var data-var='time'>21:50</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'> 2</var>, <var data-var='time'>21:47</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar <var data-var='date'> 2</var>, <var data-var='time'>21:39</var> UTC Identified - The issue has been identified and a fix is being implemented.Mar <var data-var='date'> 2</var>, <var data-var='time'>21:19</var> UTC Investigating - We're currently investigating issues with the Machines API. Customer deployments and the Fly dashboard may be affected.

Issues with the Machines API

Feb <var data-var='date'>27</var>, <var data-var='time'>20:21</var> UTC Resolved - This incident has been resolved. All platform and API operations are working normally.Feb <var data-var='date'>27</var>, <var data-var='time'>20:05</var> UTC Monitoring - API and platform operations have normalized. We are continuing to monitor to ensure full and stable recovery. Background jobs are almost fully caught up. Users may still see slightly slower requests creating new apps / orgs, but they should complete successfully. Sprite and MPG cluster creations are processing as normal.Feb <var data-var='date'>27</var>, <var data-var='time'>19:41</var> UTC Update - A second fix has been deployed and database load has returned to normal, resulting in API response times beginning to normalize. Most Machines API requests should succeed as normal, and deploys to existing apps should also work. We are working through a backlog of background jobs. New app / organization creations and other other operations that use these will continue to see increased latency or failures while we work thorough these. New MPG cluster and new Sprite creation continues to be impacted.Feb <var data-var='date'>27</var>, <var data-var='time'>19:23</var> UTC Update - An initial fix has been deployed and we are seeing improvements in load and API performance. Some operations that rely on the Graphql API, such as new app creations and some deployments, will continue to fail at this time. We are continuing to work on restoring full availability.Feb <var data-var='date'>27</var>, <var data-var='time'>19:05</var> UTC Update - We are currently seeing full API failures for requests to our Graphql API and elevated failures for the machines API. Direct calls to these apis may fail, along with many flyctl commands. We have identified the cause of the issue and are continuing to work on a fix. Existing running machines and apps should continue to be reachable, but creates, deploys, or other features relying on platform API calls will fail at this time.Feb <var data-var='date'>27</var>, <var data-var='time'>18:59</var> UTC Update - New Sprite creations are also timing out or failing at this time. We are continuing to work on a fix for this issue.Feb <var data-var='date'>27</var>, <var data-var='time'>18:53</var> UTC Update - We are continuing to work on a fix for this issue.Feb <var data-var='date'>27</var>, <var data-var='time'>18:52</var> UTC Identified - We have identified the cause of the increased latency and are working on a fix. The most common errors we are seeing is timeouts when users attempt to perform an action against a newly created app / machine resource. Those may timeout or fail with an `app|machine not found` errorFeb <var data-var='date'>27</var>, <var data-var='time'>18:50</var> UTC Investigating - We are investigating increased in API request latency and timeouts with the main platform API. This is impacting multiple operations, including creating, querying or performing actions against machines, as well as platform level operations like adding payment methods.

Is Fly.io down?

Fly.io status is UP

Lastest published events

Metrics currently experiencing issues

GraphQL API / Dashboard Issues

SJC Network Maintenance

Fly.io Services, Components and Regions

Recent Fly.io events

Metrics currently experiencing issues

GraphQL API / Dashboard Issues

SJC Network Maintenance

Low Capacity in SIN and AMS regions

Low capacity in IAD

Machine Creates Failing in ORD Region

Get your FREE account now,
5 monitors included!

Is Fly.io down?

Fly.io status is UP

Lastest published events

Metrics currently experiencing issues

GraphQL API / Dashboard Issues

SJC Network Maintenance

Fly.io Services, Components and Regions

Recent Fly.io events

Metrics currently experiencing issues

GraphQL API / Dashboard Issues

SJC Network Maintenance

Low Capacity in SIN and AMS regions

Low capacity in IAD

Machine Creates Failing in ORD Region

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!