Aug <var data-var='date'>18</var>, <var data-var='time'>08:25</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>18</var>, <var data-var='time'>08:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>28</var>, <var data-var='time'>08:02</var> UTC Scheduled - Essential maintenance work is being carried out on the Flagsmith SaaS database. The maintenance will last for approximately 30min and during this time it will not be possible to make changes to your flags from the Flagsmith dashboard. The API endpoints used by our SDKs will continue to function as normal, although there may be certain periods during the window for any projects still running on the Core API where new identities/traits will not be persisted. This will not affect the responses provided by the API.

Production Database Maintenance

Mar <var data-var='date'> 3</var>, <var data-var='time'>17:00</var> UTC Resolved - Environment documents were being rendered with numeric values cast as numbers in the JSON responses, instead of the previous behaviour where the values were returned as strings. This caused issues for statically typed languages in particular.

Numeric values incorrectly being converted to numbers in environment documents

Jan <var data-var='date'>22</var>, <var data-var='time'>18:40</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>22</var>, <var data-var='time'>18:02</var> UTC Monitoring - The latest update has resolved the issue in all regions. We are continuing to monitor. Jan <var data-var='date'>22</var>, <var data-var='time'>17:51</var> UTC Identified - The applied fix has not resolved the issue for all customers. New version being rolled out now with a fix for all customers. Jan <var data-var='date'>22</var>, <var data-var='time'>17:30</var> UTC Monitoring - A fix has been deployed to all regions. We have resolved the failures and are monitoring further impact. Jan <var data-var='date'>22</var>, <var data-var='time'>16:56</var> UTC Identified - We are receiving reports of failing Edge API requests for a small subset of Flagsmith environments. We have identified the problem and the fix is underway.

Data parsing issues in Edge API

Oct <var data-var='date'>23</var>, <var data-var='time'>14:49</var> UTC Resolved - All issues with the asynchronous task processor have been resolved and all operations in the Flagsmith UI are working as expected. Oct <var data-var='date'>23</var>, <var data-var='time'>10:26</var> UTC Monitoring - We have resolved the issue with the task processor and are no longer seeing issues with the Flagsmith UI. We are continuing to monitor performance of the task processor. Oct <var data-var='date'>23</var>, <var data-var='time'>10:08</var> UTC Investigating - We are currently investigating an issue with the Flagsmith asynchronous task processor, impacting flag change events, webhooks and audit logs in the Flagsmith UI.

Task processor down

Jul <var data-var='date'>23</var>, <var data-var='time'>21:17</var> UTC Resolved - The issue was caused by a spike of unusual traffic. This traffic has been mitigated and the issue is resolved. Jul <var data-var='date'>23</var>, <var data-var='time'>21:01</var> UTC Investigating - We're seeing a delay in processing tasks by our asynchronous task processor, meaning that changes made in the dashboard may take longer to reflect in the API.

Delayed asynchronous task processing

Jun <var data-var='date'>18</var>, <var data-var='time'>19:43</var> UTC Resolved - The incident on the third party system was resolved at 15.11 UTC. Jun <var data-var='date'>18</var>, <var data-var='time'>13:52</var> UTC Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>18</var>, <var data-var='time'>13:52</var> UTC Monitoring - It looks like the issue has been resolved on the provider's side. We are monitoring and validating. Jun <var data-var='date'>18</var>, <var data-var='time'>13:47</var> UTC Update - It appears that the issue is related to our third party hosting provider for the Flagsmith dashboard. We are following up with them. Jun <var data-var='date'>18</var>, <var data-var='time'>13:46</var> UTC Investigating - We are currently investigating alerts that app.flagsmith.com is not loading.

Flagsmith dashboard not loading

Jun <var data-var='date'> 9</var>, <var data-var='time'>10:13</var> UTC Resolved - Our website was down momentarily this morning, but it did not affect our main application (app.flagsmith.com). Users were unable to view our marketing page due to an operational issue with management of our content management system. We'll share a post mortem in 24 hours.

Public Website Down

Dec <var data-var='date'>23</var>, <var data-var='time'>06:44</var> UTC Completed - The scheduled maintenance has been completed. Dec <var data-var='date'>23</var>, <var data-var='time'>06:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>20</var>, <var data-var='time'>14:04</var> UTC Scheduled - Essential maintenance work being carried out on the Flagsmith SaaS database. The maintenance will last for approximately 1h and during this time it will not be possible to make changes to your flags from the Flagsmith dashboard. The API endpoints used by our SDKs will continue to function as normal, although there may be certain periods during the window for any projects still running on the Core API where new identities / traits will not be persisted. This will not affect the responses provided by the API.

Dec <var data-var='date'>19</var>, <var data-var='time'>14:04</var> UTC Resolved - We have received confirmation from multiple customers in the US east region now that the issue has been resolved. Dec <var data-var='date'>18</var>, <var data-var='time'>18:56</var> UTC Monitoring - Our infrastructure provider has confirmed they were intermittently unavailable between December 14, 01:00 UTC and December 17, 22:50 UTC. We are working with affected customers to confirm that this issue is resolved. Dec <var data-var='date'>17</var>, <var data-var='time'>15:49</var> UTC Identified - The issue has been isolated to a specific component in our hosting provider's network infrastructure. We are working with them, and escalating to get the issue resolved. Dec <var data-var='date'>16</var>, <var data-var='time'>23:43</var> UTC Update - We have passed on all of the troubleshooting information to our infrastructure provider and are awaiting further information from them on this issue. If you are affected by this issue, please get in touch with us at support@flagsmith.com with any information that you can share so we can identify this intermittent issue. Dec <var data-var='date'>16</var>, <var data-var='time'>21:59</var> UTC Investigating - We have had further reports of similar issues following the migration of traffic away from us-east-2, we are continuing to investigate. Dec <var data-var='date'>16</var>, <var data-var='time'>20:17</var> UTC Monitoring - We have migrated all traffic away from us-east-2 and are monitoring the impact. Dec <var data-var='date'>16</var>, <var data-var='time'>19:10</var> UTC Update - The issue seems to only be affecting clients connecting to the us-east-2 region. We're currently redirecting traffic away from the region. Dec <var data-var='date'>16</var>, <var data-var='time'>19:07</var> UTC Investigating - We are currently investigating the Edge API sporadically timing out some requests or refusing connections.

Small number of intermittent requests to Edge API failing

Dec <var data-var='date'> 4</var>, <var data-var='time'>11:30</var> UTC Resolved - Resolved at: 15:22 UTC A bug in the validation logic traits meant that identities with traits that have a value of `""` would cause an exception, resulting in the identity not being stored correctly when updates were made. This issue was resolved at 15:22 UTC, any new requests to retrieve an identity's flags, or store their traits will correctly store those traits.

Issues persisting identities with empty trait values

Nov <var data-var='date'> 5</var>, <var data-var='time'>13:28</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'> 5</var>, <var data-var='time'>10:37</var> UTC Update - Environment creation is functional again now. Nov <var data-var='date'> 5</var>, <var data-var='time'>10:32</var> UTC Monitoring - API latency is back to normal. We are continuing to monitor. Nov <var data-var='date'> 5</var>, <var data-var='time'>10:25</var> UTC Identified - The cause is due to a migration applied to the production database taking more resources than we anticipated. The migration has been completed and the load is normalising. We are monitoring and managing the load. Currently, while this issue is ongoing, it is not possible to create new environments. Nov <var data-var='date'> 5</var>, <var data-var='time'>10:19</var> UTC Investigating - We are currently investigating this issue.

Increased API latency

Jul <var data-var='date'> 9</var>, <var data-var='time'>23:30</var> UTC Resolved - Core API and admin dashboard outage on 10th July 2024.

Core API outage

Jul <var data-var='date'> 4</var>, <var data-var='time'>08:36</var> UTC Resolved - Our Core API was overwhelmed by massive traffic spike, causing the core SQL database to become extremely slow. This led to ECS tasks failing the health checks, prompting the load balancer to start and stop new tasks, which in turn added more load to the already maxed-out database. We tried several approaches to rate limiting the source of the traffic, but eventually had to temporarily stop traffic at the load balancer for 2 minutes in order to stabilise the system. We are working on implementing AWS API Gateway to include rate limiting at the gateway level to avoid these sort of incidents in the future.

Core API Outage

Mar <var data-var='date'> 5</var>, <var data-var='time'>13:38</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 5</var>, <var data-var='time'>10:30</var> UTC Update - We are continuing to monitor for any further issues. Mar <var data-var='date'> 5</var>, <var data-var='time'>10:29</var> UTC Monitoring - We have rolled back a recent change which has cleared out the backlog of tasks. Updates to flags should be propagated as normal. Mar <var data-var='date'> 5</var>, <var data-var='time'>10:27</var> UTC Investigating - We have been alerted to an issue with our asynchronous task processor which handles replicating flag updates across our network. We are currently investigating.

Issues with flag updates

Feb <var data-var='date'>20</var>, <var data-var='time'>17:30</var> UTC Completed - The scheduled maintenance has been completed. Feb <var data-var='date'>14</var>, <var data-var='time'>07:19</var> UTC Verifying - Performance issus have been resolved and we are continuing to monitor the situation. Feb <var data-var='date'>14</var>, <var data-var='time'>07:03</var> UTC Update - We are continuing to investigate. Feb <var data-var='date'>14</var>, <var data-var='time'>07:03</var> UTC Update - Maintenance is completed and the service is fully operational again, however, we are investigating lower performance on the upgraded database. Feb <var data-var='date'>14</var>, <var data-var='time'>06:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'> 9</var>, <var data-var='time'>10:21</var> UTC Scheduled - We will be performing some maintenance on the core Flagsmith database on Wednesday 14th February at 6am UTC. During this maintenance we will be upgrading the Postgres version from v11 to v15. This maintenance is necessary as v11 is now EOL as well as enabling us to complete further upgrade work on the Flagsmith application to improve functionality and performance. The maintenance will last for approximately 1h and during this time it will not be possible to make changes to your flags from the Flagsmith dashboard. The API endpoints used by our SDKs will continue to function as normal, although there may be certain periods during the window for any projects still running on the Core API where new identities / traits will not be persisted. This will not affect the responses provided by the API.

Jan <var data-var='date'>18</var>, <var data-var='time'>15:50</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>18</var>, <var data-var='time'>15:43</var> UTC Update - We are deploying the permanent fix now. Jan <var data-var='date'>18</var>, <var data-var='time'>15:36</var> UTC Update - We are continuing to work on a fix for this issue. Jan <var data-var='date'>18</var>, <var data-var='time'>15:34</var> UTC Update - We have rolled back in certain affected regions and have completed the work for the permanent fix. This is in the final stages of testing now and will be rolled out imminently. Jan <var data-var='date'>18</var>, <var data-var='time'>15:15</var> UTC Identified - We have identified the remaining issue and are implementing a fix. ETA for full resolution: 15 minutes. Jan <var data-var='date'>18</var>, <var data-var='time'>15:06</var> UTC Update - We are continuing to investigate this issue. Jan <var data-var='date'>18</var>, <var data-var='time'>15:06</var> UTC Investigating - Issues are still persisting for integrations using the Go client. We are investigating further. Jan <var data-var='date'>18</var>, <var data-var='time'>14:52</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jan <var data-var='date'>18</var>, <var data-var='time'>14:44</var> UTC Identified - The issue has been identified and a fix is being implemented. Jan <var data-var='date'>18</var>, <var data-var='time'>14:40</var> UTC Investigating - We are currently investigating this issue.

Increased error rates on the Edge API

Dec <var data-var='date'> 1</var>, <var data-var='time'>15:29</var> UTC Resolved - The incident was related to an erroneous DNS change. This has now been reverted and service should be back up and running. There may be a period of time where failures are still seen as we wait for DNS caching to be propagated. Dec <var data-var='date'> 1</var>, <var data-var='time'>15:10</var> UTC Investigating - We are currently investigating reports of a service outage on all of our infrastructure.

Service Outage

Sep <var data-var='date'>12</var>, <var data-var='time'>12:15</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'>12</var>, <var data-var='time'>11:35</var> UTC Update - We are continuing to monitor for any further issues. Sep <var data-var='date'>12</var>, <var data-var='time'>11:26</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'>12</var>, <var data-var='time'>11:14</var> UTC Investigating - We are currently investigating this issue.

Identity integrations are not being triggered

Sep <var data-var='date'> 5</var>, <var data-var='time'>12:01</var> UTC Resolved - This incident has been resolved. A postmortem will follow. Sep <var data-var='date'> 5</var>, <var data-var='time'>10:25</var> UTC Update - We are continuing to monitor for any further issues. Sep <var data-var='date'> 5</var>, <var data-var='time'>10:25</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'> 5</var>, <var data-var='time'>09:51</var> UTC Identified - We have identified a database migration that has failed as part of a new release. We are working to re-apply the migration. Sep <var data-var='date'> 5</var>, <var data-var='time'>09:45</var> UTC Investigating - We are currently investigating this issue.

Core API is not responding

Aug <var data-var='date'>10</var>, <var data-var='time'>12:54</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>10</var>, <var data-var='time'>12:26</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>10</var>, <var data-var='time'>12:15</var> UTC Identified - We were alerted to a slow DB query in newly released code at 13:09 BST. We are reverting the code and expect to be back to normal latency in the next 10 minutes. Edge API is not affected.

Performance impacted to our Core API

Jul <var data-var='date'>19</var>, <var data-var='time'>10:19</var> UTC Resolved - This incident has been resolved. We will publish a full post-mortem imminently. Jul <var data-var='date'>19</var>, <var data-var='time'>09:12</var> UTC Monitoring - We have deployed an update which has resumed consumption of the task queue. We are now processing the task queue and expect to be caught up in the next hour. Jul <var data-var='date'>19</var>, <var data-var='time'>08:02</var> UTC Identified - We have identified a database lock that has caused this issue with the task processor. We are working on an interim fix as we identify the root cause. Jul <var data-var='date'>19</var>, <var data-var='time'>06:35</var> UTC Update - We are continuing to investigate this issue with the utmost priority. Jul <var data-var='date'>19</var>, <var data-var='time'>02:58</var> UTC Investigating - At the moment, we are conducting an investigation, which indicates that any flag changes made in the last approximately two hours may not be visible to the client.

We are currently encountering difficulties with our task processing system.

Jul <var data-var='date'>12</var>, <var data-var='time'>16:17</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>12</var>, <var data-var='time'>15:56</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>12</var>, <var data-var='time'>14:25</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'>12</var>, <var data-var='time'>13:47</var> UTC Update - We are continuing to investigate this issue. Jul <var data-var='date'>12</var>, <var data-var='time'>13:46</var> UTC Investigating - We are investigating an issue where project admin users do not receive the inherited permissions on each environment in the project.

Issue affecting permissions for project admin users

Jul <var data-var='date'>10</var>, <var data-var='time'>12:58</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>10</var>, <var data-var='time'>12:50</var> UTC Monitoring - The downstream service has been successfully removed. Response times have returned to normal. We are continuing to monitor the situation. Jul <var data-var='date'>10</var>, <var data-var='time'>12:44</var> UTC Identified - We have identified an issue caused by a downstream service which is causing a knock on effect to our performance. We are currently deploying a change to remove the downstream service. Jul <var data-var='date'>10</var>, <var data-var='time'>12:29</var> UTC Investigating - We are currently investigating this issue.

Slow response times for Edge API requests

Jul <var data-var='date'> 4</var>, <var data-var='time'>10:30</var> UTC Resolved - Customer reports of erroneous flag values being served in local evaluation mode.

Erroneous flag values

Jun <var data-var='date'>16</var>, <var data-var='time'>18:13</var> UTC Resolved - Issue has been resolved. Integrations all tested as working. Post-Mortem report to come next week. Jun <var data-var='date'>16</var>, <var data-var='time'>17:40</var> UTC Identified - We are aware of issues firing environment based integrations in our Edge API. We have identified a fix and are working on the testing now. Performance of the Edge API itself is not affected.

Is Flagsmith down?

Flagsmith status is UP

Lastest published events

Production Database Maintenance

Numeric values incorrectly being converted to numbers in environment documents

Data parsing issues in Edge API

Flagsmith Services, Components and Regions

Recent Flagsmith events

Production Database Maintenance

Numeric values incorrectly being converted to numbers in environment documents

Data parsing issues in Edge API

Task processor down

Delayed asynchronous task processing

Flagsmith dashboard not loading