Jul <var data-var='date'>30</var>, <var data-var='time'>23:00</var> PDT Resolved - Connections established through the Keeper Router and Gateway in the EU region is failing. Additional details will be posted in the postmortem report.

EU KeeperPAM router and gateway connections

Jul <var data-var='date'> 8</var>, <var data-var='time'>12:14</var> PDT Resolved - API errors have been resolved. The Keeper Gateway and KSM client version is no longer returning an error. Jul <var data-var='date'> 8</var>, <var data-var='time'>11:51</var> PDT Identified - We are aware of Keeper Gateway and KSM connection errors due to "invalid client version" responses. We have identified the issue and it will be resolved in a few minutes.

Resolved: Keeper Gateway and KSM API errors

Jun <var data-var='date'>26</var>, <var data-var='time'>16:26</var> PDT Resolved - The issue is resolved. Jun <var data-var='date'>26</var>, <var data-var='time'>15:43</var> PDT Identified - Maintenance updates in the US GovCloud region are in progress which are causing unexpected API errors. The issues will be resolved shortly.

GovCloud region maintenance

Jun <var data-var='date'>19</var>, <var data-var='time'>16:00</var> PDT Resolved - The issue was resolved at 12:30PM PST. Jun <var data-var='date'>19</var>, <var data-var='time'>08:05</var> PDT Identified - We are aware of an issue with direct record sharing for enterprise users as a result of yesterday's Vault 18.3 update. We will issue a patch before 12PM PST. To work around the issue, you can click the "Can share to users outside the enterprise" option in Role enforcements, or just use the Desktop App.

Direct record sharing enforcement issue

Mar <var data-var='date'>30</var>, <var data-var='time'>20:00</var> PDT Resolved - On March 30 at 8:00 PM PST, Keeper performed a scheduled minor upgrade to our multi-region AWS RDS clusters as part of routine maintenance. The upgrade itself completed in approximately 3 minutes, followed by a standard application restart process that took about 15 minutes. While this process is typically non-disruptive, recent changes to service health checks caused unexpected customer impact during the restart window. This maintenance was not communicated in advance due to an internal miscommunication and the expectation of no impact. We apologize for the disruption and are taking steps to ensure all future maintenance—regardless of expected impact—is communicated ahead of time, as well as reviewing our health check configurations to prevent similar issues.

API errors during maintenance operation

Mar <var data-var='date'>23</var>, <var data-var='time'>14:45</var> PDT Resolved - The issue has been resolved. See postmortem page with additional details. Mar <var data-var='date'>23</var>, <var data-var='time'>13:03</var> PDT Monitoring - A fix has been implemented and we are monitoring the KeeperPAM connection stability. We will update this case with additional details soon. Mar <var data-var='date'>23</var>, <var data-var='time'>11:47</var> PDT Identified - We have identified the cause of the KeeperPAM connection errors which are related to a networking issue within the AWS ECS environment. We are actively troubleshooting with the AWS support team and will update the ticket as soon as there is a status update. Mar <var data-var='date'>23</var>, <var data-var='time'>10:07</var> PDT Investigating - We are investigating errors in establishing KeeperPAM connections in the US Data Center.

Resolved: KeeperPAM Connection Errors in US Data Center

Mar <var data-var='date'>16</var>, <var data-var='time'>11:52</var> PDT Resolved - The issue has been resolved. We will update the ticket with additional details shortly. Mar <var data-var='date'>16</var>, <var data-var='time'>11:25</var> PDT Identified - We have identified an issue causing KeeperPAM connection errors in the US Data Center. It will be resolved shortly.

Feb <var data-var='date'>25</var>, <var data-var='time'>13:00</var> PST Completed - The scheduled maintenance has been completed. Feb <var data-var='date'>25</var>, <var data-var='time'>12:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'>24</var>, <var data-var='time'>15:38</var> PST Scheduled - We will be performing scheduled maintenance in the AU data center at 16:00 UTC / 03:00 Thursday Sydney time. Downtime will be minimal, however we are notifying customers for a potential 1 hour time period.

Scheduled maintenance in AU data center

Feb <var data-var='date'>18</var>, <var data-var='time'>21:30</var> PST Resolved - As part of our planned GovCloud region maintenance involving database upgrades, restarts of the Redshift cluster were required. As a result, downstream services required configuration updates and all services are now restored.

US GovCloud Region - Login Errors Resolved

Feb <var data-var='date'>18</var>, <var data-var='time'>20:30</var> PST Completed - The scheduled maintenance has been completed. Feb <var data-var='date'>18</var>, <var data-var='time'>20:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'>18</var>, <var data-var='time'>08:32</var> PST Scheduled - We will be undergoing required maintenance in the US GovCloud environment tonight at 8PM PST. While we have allocated 30 minutes, the update should only cause a few minutes of disruption.

US GovCloud region - maintenance planned tonight

Dec <var data-var='date'> 2</var>, <var data-var='time'>16:47</var> PST Resolved - The issue has been resolved. We will update the postmortem report shortly. Dec <var data-var='date'> 2</var>, <var data-var='time'>14:48</var> PST Identified - During scheduled system updates, we have encountered unexpected NGINX 503 / 502 errors in the EU region. As of 4:50 PM CST, work is still in progress. During this time, users in the EU region will experience brief or intermittent access interruptions when using Keeper.

EU data center login errors - Resolved

Oct <var data-var='date'>20</var>, <var data-var='time'>13:31</var> PDT Resolved - Clearing this alert as most AWS services are restored. Oct <var data-var='date'>20</var>, <var data-var='time'>10:25</var> PDT Identified - There are intermittent connectivity issues due to the AWS outage in US-EAST-1 affecting KeeperPAM cloud connections, scheduled rotations and discovery operations.

AWS outage affecting KeeperPAM connections and scheduled rotations

Oct <var data-var='date'>20</var>, <var data-var='time'>08:17</var> PDT Resolved - All Keeper email delivery services in US-EAST-1 were restored around 4AM PST. Oct <var data-var='date'>20</var>, <var data-var='time'>03:00</var> PDT Identified - SMS delivery services have been restored, but AWS Lambda is still down which affects our email delivery of device approvals. We recommend using the "Keeper Push" or TOTP method of device approval. Oct <var data-var='date'>20</var>, <var data-var='time'>02:20</var> PDT Investigating - AWS is reporting errors in the US-EAST region that are affecting multiple services, affecting email and SMS delivery which impacts Keeper device approvals. The AWS Service Health dashboard can be monitored here: https://health.aws.amazon.com/health/status

Email delivery in US region due to AWS US-EAST outage

Sep <var data-var='date'>10</var>, <var data-var='time'>08:49</var> PDT Resolved - The issue with Auth API errors in the EU region has been resolved. Additional details will be posted in the postmortem shortly. Sep <var data-var='date'>10</var>, <var data-var='time'>08:42</var> PDT Identified - We are aware of Auth API errors in EU region and we have identified the isssue. The issue will be resolved shortly.

EU Region login API errors

Aug <var data-var='date'>21</var>, <var data-var='time'>10:36</var> PDT Resolved - Resolved - At 9:49 AM PST, we experienced login API request failures due to errors on an RDS database in the US region. The database failover completed successfully, and all systems were fully operational by 10:00 AM PST.

Resolved: Vault login API errors

Jul <var data-var='date'> 6</var>, <var data-var='time'>08:00</var> PDT Resolved - Several customers have been experiencing SSO login errors this morning. The team determined the cause of the issue which is detailed in the postmortem report.

SSO login errors with strict SP cert matching

Jun <var data-var='date'>24</var>, <var data-var='time'>09:01</var> PDT Resolved - The issue has been resolved. The outage was caused by a high volume of database locks related to a large BreachWatch data update. The engineering team is tracking down the root cause and will be addressing a preventative solution today. Jun <var data-var='date'>24</var>, <var data-var='time'>08:28</var> PDT Update - We are continuing to investigate this issue. Jun <var data-var='date'>24</var>, <var data-var='time'>08:27</var> PDT Investigating - We are investigating vault login errors

Vault login errors - resolved

Jun <var data-var='date'>11</var>, <var data-var='time'>05:46</var> PDT Resolved - Login errors in the EU region have been fully resolved. We will update this incident postmortem with additional information shortly. Jun <var data-var='date'>11</var>, <var data-var='time'>05:16</var> PDT Identified - We have identified the issue causing login errors in the EU data center. Our team is working on a resolution. Jun <var data-var='date'>11</var>, <var data-var='time'>05:04</var> PDT Investigating - We are investigating login errors in the EU data center.

Login errors in the EU data center

Mar <var data-var='date'>13</var>, <var data-var='time'>14:15</var> PDT Resolved - The issue has been resolved. Additional information is in the postmortem report. Mar <var data-var='date'>13</var>, <var data-var='time'>12:17</var> PDT Identified - Some users are receiving errors when downloading file attachments. We are working on resolution.

File attachment download errors for new uploaded files

Mar <var data-var='date'>10</var>, <var data-var='time'>17:00</var> PDT Resolved - Between 5:15PM PST and 5:30PM PST, there were a high number of Vault Login errors.

Vault login errors

Mar <var data-var='date'> 6</var>, <var data-var='time'>10:25</var> PST Resolved - The issue has been resolved Mar <var data-var='date'> 6</var>, <var data-var='time'>10:24</var> PST Update - We are continuing to investigate this issue. Mar <var data-var='date'> 6</var>, <var data-var='time'>10:23</var> PST Investigating - We are investigating login errors and our team is currently working on the issue.

Vault Login errors - resolved

Feb <var data-var='date'>23</var>, <var data-var='time'>21:05</var> PST Resolved - The issue has been resolved and root cause determined. We will update the postmortem on this issue shortly. Feb <var data-var='date'>23</var>, <var data-var='time'>20:48</var> PST Identified - We are still working on the issue which is causing login errors in the US region. Please use Keeper's "work offline" while we are working to resolve the issue.

Keeper vault login errors - resolved

Feb <var data-var='date'>23</var>, <var data-var='time'>20:28</var> PST Resolved - The issue has been resolved. Feb <var data-var='date'>23</var>, <var data-var='time'>19:54</var> PST Investigating - We are currently investigating login errors in the US region.

Login errors in US region

Feb <var data-var='date'>10</var>, <var data-var='time'>09:49</var> PST Resolved - We have resolved the issue and plan to deploy a backend update shortly. Feb <var data-var='date'>10</var>, <var data-var='time'>09:30</var> PST Identified - We are aware of a high number of errors in the US region. The issue will be fixed shortly.

Jan <var data-var='date'>28</var>, <var data-var='time'>20:30</var> PST Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>28</var>, <var data-var='time'>20:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>28</var>, <var data-var='time'>14:37</var> PST Scheduled - We will be performing system maintenance at 8PM PST. APIs may return errors for approximately 2 minutes.

Is KeeperSecurity down?

KeeperSecurity status is UP

Lastest published events

EU KeeperPAM router and gateway connections

Resolved: Keeper Gateway and KSM API errors

GovCloud region maintenance

KeeperSecurity Services, Components and Regions

Recent KeeperSecurity events

EU KeeperPAM router and gateway connections

Resolved: Keeper Gateway and KSM API errors

GovCloud region maintenance

Direct record sharing enforcement issue

API errors during maintenance operation

Resolved: KeeperPAM Connection Errors in US Data Center