Oct <var data-var='date'> 1</var>, <var data-var='time'>09:20</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'>29</var>, <var data-var='time'>19:30</var> UTC Identified - Customers may be unable to find newly-created items in Konnect using the Search function. Additionally, analytics and gateway manager filtering by newly-created objects may also be unavailable.

Search ingestion delayed in US Region

Sep <var data-var='date'>18</var>, <var data-var='time'>20:00</var> UTC Completed - The scheduled maintenance has been completed. Sep <var data-var='date'>18</var>, <var data-var='time'>18:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>14</var>, <var data-var='time'>21:23</var> UTC Update - We will be undergoing scheduled maintenance during this time. Sep <var data-var='date'>14</var>, <var data-var='time'>21:22</var> UTC Scheduled - We will be performing scheduled maintenance to upgrade the Konnect database infrastructure in Singapore region. This update will improve the overall reliability and performance of the platform. Expected Impact: - Maintenance will occur in SG experiencing a brief (up to 2 minutes) control plane downtime. - During this time, configuration or policy updates in Konnect may not apply until the maintenance completes in that region. - Gateway data plane traffic will not be affected and will continue to process requests normally. We appreciate your understanding as we continue to enhance the stability and resilience of the Konnect platform.

Konnect Database Infrastructure Maintenance [SG]

Aug <var data-var='date'>20</var>, <var data-var='time'>14:15</var> UTC Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>20</var>, <var data-var='time'>13:15</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>20</var>, <var data-var='time'>00:01</var> UTC Scheduled - We will be performing scheduled security maintenance across all Konnect regions. This maintenance is part of our ongoing efforts to maintain the security and reliability of the platform. Expected Impact: * Maintenance will be performed across all production regions. * Customers may experience brief periods of degraded Control Plane functionality while the maintenance is in progress. * Gateway Data Plane traffic will not be affected and will continue to process requests normally. We appreciate your understanding as we continue to enhance the security, stability, and resilience of the Konnect platform.

Scheduled Security Maintenance – All Regions

Aug <var data-var='date'>20</var>, <var data-var='time'>01:16</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>19</var>, <var data-var='time'>20:39</var> UTC Identified - Analytics queries may be unavailable or returning incomplete data in EU.

Analytics queries may be unavailable or returning incomplete data in EU.

Aug <var data-var='date'>17</var>, <var data-var='time'>22:10</var> UTC Resolved - From 21:33z to 21:51z the Konnect Control Plane had an elevated error rate when updating some entities. This was due to a misconfiguration in our encryption policy evaluation that temporarily rejected all fields that required encryption.

Elevated error rates from control plane APIs in US Region

Aug <var data-var='date'>12</var>, <var data-var='time'>06:00</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>11</var>, <var data-var='time'>14:31</var> UTC Investigating - Some customers may notice some missing data from charts and graphs while this lag occurs. The team is working to fix the processing and back-fill any missing data.

Some analytics ingestion is lagging in US prod

Aug <var data-var='date'> 1</var>, <var data-var='time'>00:07</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>31</var>, <var data-var='time'>23:42</var> UTC Monitoring - Infrastructure maintenance on the US region is complete. We will continue to monitor this region to ensure stability of the platform. Jul <var data-var='date'>31</var>, <var data-var='time'>23:25</var> UTC Investigating - We will be performing maintenance. Customers may experience brief intermittent service disruptions or increased latency. We are working to minimize any impact and will provide updates here as the maintenance progresses. We apologize for any inconvenience and appreciate your patience. Expected Impact: - Maintenance will occur in the US region only. - During this time, configuration or policy updates in Konnect may be delayed until the maintenance completes in that region. - Gateway data plane traffic will not be affected and will continue to process requests normally. We do not anticipate customer impact, but brief transient connectivity disruptions may occur while changes are being applied.

Performing Infrastructure Maintenance - US Region

Jul <var data-var='date'>24</var>, <var data-var='time'>13:30</var> UTC Resolved - We identified a brief period of elevated API errors in the EU region, from 13:21 - 13:23 UTC, resulting in HTTP 5xx responses across multiple Konnect services and potentially impacting actions taken on the Konnect UI. The underlying cause is still under investigation.

Instability across Konnect services in EU

Jul <var data-var='date'>13</var>, <var data-var='time'>09:10</var> UTC Resolved - This issue has been resolved. Jul <var data-var='date'>13</var>, <var data-var='time'>08:44</var> UTC Update - We are continuing to investigate the issue. Jul <var data-var='date'>13</var>, <var data-var='time'>07:31</var> UTC Update - We are continuing to investigate the issue. Jul <var data-var='date'>13</var>, <var data-var='time'>07:10</var> UTC Investigating - Login to Konnect intermittently failing. Customers can retry login in order to gain access to the platform. API calls to Konnect are succeeding.

Konnect UI Login Degraded

Jul <var data-var='date'>10</var>, <var data-var='time'>02:41</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>10</var>, <var data-var='time'>01:55</var> UTC Investigating - Customers are unable to visualize the platform usage analytics data. This does not impact data ingestion.

Konnect Observability UI Experience Degraded

Jul <var data-var='date'> 2</var>, <var data-var='time'>14:01</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 2</var>, <var data-var='time'>13:56</var> UTC Monitoring - We are monitoring an incident that resulted in an increased error rate in the Portal APIs. A fix has been applied, and error rates have decreased.

Increased error rate in the Portal APIs

Jul <var data-var='date'> 2</var>, <var data-var='time'>09:00</var> UTC Resolved - The Konnect control plane in the India was incorrectly rejecting all user tokens between 08:30am UTC to 9:08am UTC. A misconfiguration of the token validation URL was deployed, noticed by our monitoring, and rolled back. No dataplane traffic or proxy behavior would have been impacted. Impacts were: - API calls, including from tools like DecK and Terraform would receive 401 responses - Users on the Konnect UI may have been logged out if the UI made any control plane calls that resulted in a 401 response

Authentication issue in IN Region for Konnect Control Plane

Jun <var data-var='date'>29</var>, <var data-var='time'>15:56</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'> 2</var>, <var data-var='time'>19:48</var> UTC Update - Konnect Middle East Region: Indefinite Outage The Konnect Middle East Region is undergoing a prolonged outage due to the operational loss of AWS datacenters in the region. The duration of this outage is undetermined. As a result, we have no timeline for service restoration in the ME Region Customers are encouraged to migrate to one of our other Konnect control plane hosting regions in the EU, Australia, India or the United States. If assistance is required, please reach out to your Kong support representative. Mar <var data-var='date'>23</var>, <var data-var='time'>23:19</var> UTC Update - We are observing increased error rates across services in the Konnect Middle East region. Data plane proxy traffic is not impacted. We are actively working with AWS to resolve the issue and will provide updates as more information becomes available. Mar <var data-var='date'>20</var>, <var data-var='time'>21:12</var> UTC Update - Customers in the Middle East (Bahrain) region are operating in a reduced Availability Zone (AZ) configuration due to an ongoing AWS infrastructure issue. The platform remains available, but some customers in this region may experience intermittent performance degradation or elevated error rates. Data plane proxy traffic is not impacted. We are actively working with AWS to resolve the issue and will provide updates as more information becomes available. Mar <var data-var='date'> 5</var>, <var data-var='time'>13:16</var> UTC Update - It was discovered that DNS for KIC reconciliation endpoints were still pointing to the out-of-service region. We have failed over these DNS records now to properly point to our new region. Customers using KIC would have seen 500s' during the reconciliation to Konnect. This was resolved at 10:30 UTC on March 5th. Mar <var data-var='date'> 3</var>, <var data-var='time'>08:53</var> UTC Update - Current Status Summary for Konnect Middle East Region: All services have been moved to our failover region in Middle East Almost all services are operational, with 2 caveats: - Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service. - The Debugger is operational including trace data; however session logs will not be available in this region. - Observability is operational; however UUID to display name resolution will not be available in this region. Everything else is back online. The team will update as we make progress on these remaining items. Mar <var data-var='date'> 3</var>, <var data-var='time'>05:35</var> UTC Update - Current Status Summary for Konnect Middle East Region: All services have been moved to our failover region in Middle East Almost all services are operational, with 2 caveats: - Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service. - Debugger is unable to accept incoming trace data. While debugging sessions can be started, none of the trace data will be persisted in Konnect until the datastore for traces is restored. Everything else is back online. The team will update as we make progress on these remaining items. Mar <var data-var='date'> 3</var>, <var data-var='time'>05:13</var> UTC Update - The data store for traces sent to Debugger is still experiencing issues in the Middle East region. Debugging sessions can be started but no spans will be able to be ingested. The team will update as they bring that service online. Mar <var data-var='date'> 3</var>, <var data-var='time'>03:45</var> UTC Update - Service remains restored, and Kong engineers will continue to monitor the apps in the new region. Mar <var data-var='date'> 3</var>, <var data-var='time'>03:03</var> UTC Update - User permission data is now propagating normally again. Mar <var data-var='date'> 2</var>, <var data-var='time'>22:45</var> UTC Update - Status Update (PDP – ME-SOUTH-1) User permission data replication in ME-SOUTH-1 (Bahrain) region are experiencing extended latencies. Mitigation has been implemented to cap this to 15 minutes. Updated data should now propagate within a maximum of 15 minutes. We continue to monitor the situation and will provide further updates as needed. Mar <var data-var='date'> 2</var>, <var data-var='time'>21:50</var> UTC Update - The following services have been restored: - All Konnect UI-accessible products except for Event Gateway and Metering - Dataplane connections to the Konnect Control Plane for configuration download and analytics upload Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service. Mar <var data-var='date'> 2</var>, <var data-var='time'>21:39</var> UTC Monitoring - The Kong engineering team is now verifying restored services in the Middle East region Mar <var data-var='date'> 2</var>, <var data-var='time'>21:35</var> UTC Update - We have restored partial service to Konnect. The UI is working again for Gateway Manager and Dev Portal, as well as all backing APIs. Dataplane connections to the Control Plane and Telemetry endpoints have been restored for 3 hours and should continue to work. Mar <var data-var='date'> 2</var>, <var data-var='time'>20:05</var> UTC Update - We continue to make progress on restoring to a failover region for the Middle East region. Key AWS services to support our failover efforts are beginning to work as of 11am PST. The Konnect engineering team is working to restore and validate all services and data and will update in one hour. Please note the control plane and telemetry endpoints are already restored, so dataplanes can connect to Konnect. Mar <var data-var='date'> 2</var>, <var data-var='time'>17:36</var> UTC Update - Control Plane restored for Config Download and Telemetry We have restored the control plane and telemetry endpoints. Dataplanes can now connect to the control plane api at `me.cp0.konghq.com` and `me.tp0.konghq.com` This is a read-only instance set up for this incident. Configuration changes are still blocked. But now customers should be able to connect new dataplanes and download their configuration, as well as send telemetry. The Admin API for changing control plane configuration remains down. Mar <var data-var='date'> 2</var>, <var data-var='time'>17:26</var> UTC Update - We are actively migrating the affected services to operate on other Availability Zones in the Bahrain region (ME-SOUTH-1 Region), but critical AWS services are not responding in that region. Our engineers are working to restore full service functionality and we are communicating with AWS to attempt to determine our failover options. At this time we cannot provide an ETA. We will provide further updates as they become available. Thank you for your patience. Mar <var data-var='date'> 2</var>, <var data-var='time'>15:26</var> UTC Update - We are actively migrating the affected services to operate on the two healthy Availability Zones in the Bahrain region (ME-SOUTH-1 Region). Our engineers are working to restore full service functionality and expect the migration to be completed within the next few hours. We will provide further updates as they become available. Thank you for your patience. Mar <var data-var='date'> 2</var>, <var data-var='time'>11:27</var> UTC Update - The incident is ongoing. We are working closely with our cloud provider to monitor restoration efforts. Due to significant physical infrastructure damage in both the UAE and Bahrain regions, recovery is expected to take more time. We appreciate your patience as we work toward a full resolution. Mar <var data-var='date'> 2</var>, <var data-var='time'>06:31</var> UTC Update - We are currently observing renewed elevated error rates across multiple Konnect services in ME region related to the AWS outage in that region. Mar <var data-var='date'> 2</var>, <var data-var='time'>01:54</var> UTC Identified - We are seeing many services in ME region recovering and some services are in degraded state. Dataplanes are not affected as part of this incident. All other Konnect regions are 100% operational and healthy. Our engineers are closely monitoring the situation. We will provide additional updates as soon as more information becomes available. Mar <var data-var='date'> 1</var>, <var data-var='time'>20:07</var> UTC Investigating - We are currently observing renewed elevated 5XX error rates across multiple Konnect services in ME region. Our engineering team is actively investigating the issue and assessing customer impact. We will provide additional updates as soon as more information becomes available. Mar <var data-var='date'> 1</var>, <var data-var='time'>18:41</var> UTC Monitoring - AWS continues to experience issues some services. However most of Konnect services are now recovered and operating as expected. Our engineers are closely monitoring the services and it's operational efficiency. Mar <var data-var='date'> 1</var>, <var data-var='time'>14:45</var> UTC Identified - The increased error rates are associated with the AWS me-central-1 outage. Our engineers are monitoring the situation closely. Mar <var data-var='date'> 1</var>, <var data-var='time'>13:08</var> UTC Investigating - We are seeing increased 5xx in the ME Region. Our engineers are currently investigating the issue.

Major outage in Konnect Middle East region due to loss of mutliple availability zones

Jun <var data-var='date'>17</var>, <var data-var='time'>13:21</var> UTC Resolved - This incident has been resolved. The Konnect UI has recovered, and users should no longer experience application errors when accessing product areas in Konnect. The issue was only affecting a very small set of Konnect users. We will continue to monitor the service to ensure ongoing stability. Jun <var data-var='date'>16</var>, <var data-var='time'>21:56</var> UTC Monitoring - A fix has been implemented and our team is actively monitoring the state. Jun <var data-var='date'>16</var>, <var data-var='time'>18:43</var> UTC Update - We are continuing to investigate this issue. Jun <var data-var='date'>16</var>, <var data-var='time'>18:43</var> UTC Investigating - We are investigating reports of a small subset of users encountering application errors when accessing Konnect UI. Our team is actively investigating the issue and will provide updates as more information becomes available.

Konnect UI experiencing application error state

Jun <var data-var='date'>15</var>, <var data-var='time'>06:38</var> UTC Resolved - Customers were unable to access the Konnect UI. Upon loading the interface, an error message reading "Could not fetch user session data - Please reload the page to try again" was displayed, preventing access to the platform. This issue affected all regions globally. Control plane configurations and data plane payloads using direct API access were not impacted during this time.

Konnect UI Degradation

Jun <var data-var='date'>12</var>, <var data-var='time'>16:00</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>12</var>, <var data-var='time'>15:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'> 4</var>, <var data-var='time'>15:44</var> UTC Scheduled - We will be performing scheduled maintenance to upgrade the Konnect database infrastructure in Australia region. This update will improve the overall reliability and performance of the platform. Expected Impact: - Maintenance will occur in AU experiencing a brief (up to 2 minutes) control plane downtime. - During this time, configuration or policy updates in Konnect may not apply until the maintenance completes in that region. - Gateway data plane traffic will not be affected and will continue to process requests normally. We appreciate your understanding as we continue to enhance the stability and resilience of the Konnect platform.

Konnect Database Infrastructure Maintenance [AU]

Jun <var data-var='date'>11</var>, <var data-var='time'>15:00</var> UTC Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>11</var>, <var data-var='time'>14:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'> 9</var>, <var data-var='time'>20:08</var> UTC Scheduled - We will be performing planned network infrastructure enhancements in the US region during this maintenance window. We do not anticipate customer impact, but brief transient connectivity disruptions may occur while changes are being applied.

Scheduled Infrastructure Maintenance - US Region

Jun <var data-var='date'> 9</var>, <var data-var='time'>19:34</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'> 9</var>, <var data-var='time'>19:25</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 9</var>, <var data-var='time'>19:15</var> UTC Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'> 9</var>, <var data-var='time'>19:08</var> UTC Investigating - Investigating incorrectly filtered results from the Developers list API

Investigating incorrectly filtered results from Developers list API

May <var data-var='date'>26</var>, <var data-var='time'>17:12</var> UTC Resolved - The incident has been resolved. The cause is still being investigated and will be posted when available. May <var data-var='date'>26</var>, <var data-var='time'>15:41</var> UTC Update - We are continuing to monitor for any further issues. May <var data-var='date'>26</var>, <var data-var='time'>15:41</var> UTC Monitoring - From 14:00UTC to 15:38UTC some customers were unable to update custom plugins that required validation from the Konnect control plane. This would include schema and handler code validation. The konnect team has reloaded the validation service which allowed validations to continue again. The team is investigating the cause.

Custom Plugin Validation was failing in US region

May <var data-var='date'>20</var>, <var data-var='time'>14:25</var> UTC Resolved - This incident has been resolved. May <var data-var='date'>20</var>, <var data-var='time'>14:24</var> UTC Update - We are continuing to monitor for any further issues. May <var data-var='date'>20</var>, <var data-var='time'>14:23</var> UTC Update - A fix has been implemented and we are monitoring the results. May <var data-var='date'>20</var>, <var data-var='time'>14:23</var> UTC Monitoring - The cause has been identified. The networking load balancer was unable to reach the gateway. Dev portals are now reachable. May <var data-var='date'>20</var>, <var data-var='time'>14:04</var> UTC Investigating - Developers portals in the SG regions are not reachable. Existing app-registration and dataplane traffic continue to work. The team is investigating the cause.

Dev Portals unreachable in SG Region

May <var data-var='date'>19</var>, <var data-var='time'>21:55</var> UTC Resolved - The control plane in US continues to remain operational. The incident has been resolved. May <var data-var='date'>19</var>, <var data-var='time'>21:28</var> UTC Update - We are continuing to monitor for any further issues. May <var data-var='date'>19</var>, <var data-var='time'>21:28</var> UTC Monitoring - The Konnect control plane in the US region has recovered. The team is continuing to monitor. May <var data-var='date'>19</var>, <var data-var='time'>21:22</var> UTC Investigating - Konnect Control Plane is unavailable in the US East region. Customer dataplanes with valid configurations will remain up, but new dataplanes will not be able to pull configurations. The team is investigating.

Konnect Control Plane down in US Region

May <var data-var='date'>11</var>, <var data-var='time'>03:19</var> UTC Resolved - The issue affecting Analytics in the AU (ap-southeast-2) region has been resolved. The problem was caused by an incorrect node configuration, which has now been corrected and deployed. All services have recovered, and analytics data is returning as expected. No other regions were impacted. May <var data-var='date'>11</var>, <var data-var='time'>03:11</var> UTC Update - We are seeing analytics services in AU region recovering. May <var data-var='date'>11</var>, <var data-var='time'>03:09</var> UTC Monitoring - A fix has been deployed to the affected AU region. Analytics services are recovering, and we are actively monitoring the system to ensure stability. May <var data-var='date'>11</var>, <var data-var='time'>03:06</var> UTC Investigating - Analytics in the AU region is currently experiencing issues. No other regions are impacted at this time. Our engineering team is actively investigating the problem.

Analytics in AU region is experiencing issues

May <var data-var='date'> 8</var>, <var data-var='time'>20:00</var> UTC Resolved - Kong proactively deployed an updated license to control planes ahead of an upcoming expiration to avoid any customer disruption. As a side effect of this update, data plane nodes were incorrectly displayed as out-of-sync in the Konnect UI. The data planes themselves continued operating normally with a valid configuration throughout. The team performed a progressive rollout that triggered a full configuration sync, which cleared the inaccurate status. All nodes returned to an in-sync state shortly after. No traffic was impacted at any point. Data planes continued routing requests normally and would receive configuration updates. The only customer-visible symptom was the inaccurate out-of-sync indicator in the UI. The full configuration sync step has been added to the standard runbook for license updates, so future license rotations will not produce this false positive. Engineering is also improving the out-of-sync detection logic to reduce false positives going forward.

Some dataplanes incorrectly showing 'out of sync' in Konnect UI

Apr <var data-var='date'>10</var>, <var data-var='time'>21:11</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'>10</var>, <var data-var='time'>19:00</var> UTC Monitoring - We have rolled out the fix and will continue monitoring the situation. The system is now functioning as expected, but we are keeping a close eye on performance to ensure stability. We will provide another update if any issues arise. Apr <var data-var='date'>10</var>, <var data-var='time'>18:41</var> UTC Update - We have a fix and are rolling the change out. We will begin monitoring the situation closely and will provide another update once the rollout is complete. Apr <var data-var='date'>10</var>, <var data-var='time'>18:21</var> UTC Identified - We have identified the issue and are working on a fix Apr <var data-var='date'>10</var>, <var data-var='time'>18:07</var> UTC Investigating - Some analytics data in the US may be experiencing data loss. We are currently investigating the cause of this issue and will provide updates as they become available.

Loss of analytics payload from data plane node (US Only)

Apr <var data-var='date'> 9</var>, <var data-var='time'>18:06</var> UTC Resolved - Customers running 3.13 and below who use the OIDC plugin with `tls_client_auth_ssl_verify` unset would have seen this value change to `false` if they updated the config after the rollback of new defaults following the 3.14 release. The rollback incorrectly flipped the oidc plugin tls_client_auth_ssl_verify to false as a default, which was not one of the items recently switched to default true and has instead been defaulted true for some time. We have rolled out a fix to prod to change this default back to true. Updates to the OIDC plugin should once again keep this value set to true if not specifically defined.

Is Kong down?

Kong status is UP

Lastest published events

Search ingestion delayed in US Region

Konnect Database Infrastructure Maintenance [SG]

Scheduled Security Maintenance – All Regions

Kong Services, Components and Regions

Recent Kong events

Search ingestion delayed in US Region

Konnect Database Infrastructure Maintenance [SG]

Scheduled Security Maintenance – All Regions

Analytics queries may be unavailable or returning incomplete data in EU.

Elevated error rates from control plane APIs in US Region

Some analytics ingestion is lagging in US prod