Mar <var data-var='date'>30</var>, <var data-var='time'>20:00</var> PDT Resolved - On March 30 at 8:00 PM PST, Keeper performed a scheduled minor upgrade to our multi-region AWS RDS clusters as part of routine maintenance. The upgrade itself completed in approximately 3 minutes, followed by a standard application restart process that took about 15 minutes. While this process is typically non-disruptive, recent changes to service health checks caused unexpected customer impact during the restart window. This maintenance was not communicated in advance due to an internal miscommunication and the expectation of no impact. We apologize for the disruption and are taking steps to ensure all future maintenance—regardless of expected impact—is communicated ahead of time, as well as reviewing our health check configurations to prevent similar issues.

API errors during maintenance operation

Mar <var data-var='date'>23</var>, <var data-var='time'>14:45</var> PDT Resolved - The issue has been resolved. See postmortem page with additional details.Mar <var data-var='date'>23</var>, <var data-var='time'>13:03</var> PDT Monitoring - A fix has been implemented and we are monitoring the KeeperPAM connection stability. We will update this case with additional details soon.Mar <var data-var='date'>23</var>, <var data-var='time'>11:47</var> PDT Identified - We have identified the cause of the KeeperPAM connection errors which are related to a networking issue within the AWS ECS environment. We are actively troubleshooting with the AWS support team and will update the ticket as soon as there is a status update.Mar <var data-var='date'>23</var>, <var data-var='time'>10:07</var> PDT Investigating - We are investigating errors in establishing KeeperPAM connections in the US Data Center.

Resolved: KeeperPAM Connection Errors in US Data Center

Mar <var data-var='date'>16</var>, <var data-var='time'>11:52</var> PDT Resolved - The issue has been resolved. We will update the ticket with additional details shortly.Mar <var data-var='date'>16</var>, <var data-var='time'>11:25</var> PDT Identified - We have identified an issue causing KeeperPAM connection errors in the US Data Center. It will be resolved shortly.

Feb <var data-var='date'>25</var>, <var data-var='time'>13:00</var> PST Completed - The scheduled maintenance has been completed.Feb <var data-var='date'>25</var>, <var data-var='time'>12:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Feb <var data-var='date'>24</var>, <var data-var='time'>15:38</var> PST Scheduled - We will be performing scheduled maintenance in the AU data center at 16:00 UTC / 03:00 Thursday Sydney time. Downtime will be minimal, however we are notifying customers for a potential 1 hour time period.

Scheduled maintenance in AU data center

Feb <var data-var='date'>18</var>, <var data-var='time'>21:30</var> PST Resolved - As part of our planned GovCloud region maintenance involving database upgrades, restarts of the Redshift cluster were required. As a result, downstream services required configuration updates and all services are now restored.

US GovCloud Region - Login Errors Resolved

Feb <var data-var='date'>18</var>, <var data-var='time'>20:30</var> PST Completed - The scheduled maintenance has been completed.Feb <var data-var='date'>18</var>, <var data-var='time'>20:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Feb <var data-var='date'>18</var>, <var data-var='time'>08:32</var> PST Scheduled - We will be undergoing required maintenance in the US GovCloud environment tonight at 8PM PST. While we have allocated 30 minutes, the update should only cause a few minutes of disruption.

US GovCloud region - maintenance planned tonight

Dec <var data-var='date'> 2</var>, <var data-var='time'>16:47</var> PST Resolved - The issue has been resolved. We will update the postmortem report shortly.Dec <var data-var='date'> 2</var>, <var data-var='time'>14:48</var> PST Identified - During scheduled system updates, we have encountered unexpected NGINX 503 / 502 errors in the EU region. As of 4:50 PM CST, work is still in progress. During this time, users in the EU region will experience brief or intermittent access interruptions when using Keeper.

EU data center login errors - Resolved

Oct <var data-var='date'>20</var>, <var data-var='time'>13:31</var> PDT Resolved - Clearing this alert as most AWS services are restored.Oct <var data-var='date'>20</var>, <var data-var='time'>10:25</var> PDT Identified - There are intermittent connectivity issues due to the AWS outage in US-EAST-1 affecting KeeperPAM cloud connections, scheduled rotations and discovery operations.

AWS outage affecting KeeperPAM connections and scheduled rotations

Oct <var data-var='date'>20</var>, <var data-var='time'>08:17</var> PDT Resolved - All Keeper email delivery services in US-EAST-1 were restored around 4AM PST.Oct <var data-var='date'>20</var>, <var data-var='time'>03:00</var> PDT Identified - SMS delivery services have been restored, but AWS Lambda is still down which affects our email delivery of device approvals. We recommend using the "Keeper Push" or TOTP method of device approval.Oct <var data-var='date'>20</var>, <var data-var='time'>02:20</var> PDT Investigating - AWS is reporting errors in the US-EAST region that are affecting multiple services, affecting email and SMS delivery which impacts Keeper device approvals. The AWS Service Health dashboard can be monitored here: https://health.aws.amazon.com/health/status

Email delivery in US region due to AWS US-EAST outage

Sep <var data-var='date'>10</var>, <var data-var='time'>08:49</var> PDT Resolved - The issue with Auth API errors in the EU region has been resolved. Additional details will be posted in the postmortem shortly.Sep <var data-var='date'>10</var>, <var data-var='time'>08:42</var> PDT Identified - We are aware of Auth API errors in EU region and we have identified the isssue. The issue will be resolved shortly.

EU Region login API errors

Aug <var data-var='date'>21</var>, <var data-var='time'>10:36</var> PDT Resolved - Resolved - At 9:49 AM PST, we experienced login API request failures due to errors on an RDS database in the US region. The database failover completed successfully, and all systems were fully operational by 10:00 AM PST.

Resolved: Vault login API errors

Jul <var data-var='date'> 6</var>, <var data-var='time'>08:00</var> PDT Resolved - Several customers have been experiencing SSO login errors this morning. The team determined the cause of the issue which is detailed in the postmortem report.

SSO login errors with strict SP cert matching

Jun <var data-var='date'>24</var>, <var data-var='time'>09:01</var> PDT Resolved - The issue has been resolved. The outage was caused by a high volume of database locks related to a large BreachWatch data update. The engineering team is tracking down the root cause and will be addressing a preventative solution today.Jun <var data-var='date'>24</var>, <var data-var='time'>08:28</var> PDT Update - We are continuing to investigate this issue.Jun <var data-var='date'>24</var>, <var data-var='time'>08:27</var> PDT Investigating - We are investigating vault login errors

Vault login errors - resolved

Jun <var data-var='date'>11</var>, <var data-var='time'>05:46</var> PDT Resolved - Login errors in the EU region have been fully resolved. We will update this incident postmortem with additional information shortly.Jun <var data-var='date'>11</var>, <var data-var='time'>05:16</var> PDT Identified - We have identified the issue causing login errors in the EU data center. Our team is working on a resolution.Jun <var data-var='date'>11</var>, <var data-var='time'>05:04</var> PDT Investigating - We are investigating login errors in the EU data center.

Login errors in the EU data center

Mar <var data-var='date'>13</var>, <var data-var='time'>14:15</var> PDT Resolved - The issue has been resolved. Additional information is in the postmortem report.Mar <var data-var='date'>13</var>, <var data-var='time'>12:17</var> PDT Identified - Some users are receiving errors when downloading file attachments. We are working on resolution.

File attachment download errors for new uploaded files

Mar <var data-var='date'>10</var>, <var data-var='time'>17:00</var> PDT Resolved - Between 5:15PM PST and 5:30PM PST, there were a high number of Vault Login errors.

Vault login errors

Mar <var data-var='date'> 6</var>, <var data-var='time'>10:25</var> PST Resolved - The issue has been resolvedMar <var data-var='date'> 6</var>, <var data-var='time'>10:24</var> PST Update - We are continuing to investigate this issue.Mar <var data-var='date'> 6</var>, <var data-var='time'>10:23</var> PST Investigating - We are investigating login errors and our team is currently working on the issue.

Vault Login errors - resolved

Feb <var data-var='date'>23</var>, <var data-var='time'>21:05</var> PST Resolved - The issue has been resolved and root cause determined. We will update the postmortem on this issue shortly.Feb <var data-var='date'>23</var>, <var data-var='time'>20:48</var> PST Identified - We are still working on the issue which is causing login errors in the US region. Please use Keeper's "work offline" while we are working to resolve the issue.

Keeper vault login errors - resolved

Feb <var data-var='date'>23</var>, <var data-var='time'>20:28</var> PST Resolved - The issue has been resolved.Feb <var data-var='date'>23</var>, <var data-var='time'>19:54</var> PST Investigating - We are currently investigating login errors in the US region.

Login errors in US region

Feb <var data-var='date'>10</var>, <var data-var='time'>09:49</var> PST Resolved - We have resolved the issue and plan to deploy a backend update shortly.Feb <var data-var='date'>10</var>, <var data-var='time'>09:30</var> PST Identified - We are aware of a high number of errors in the US region. The issue will be fixed shortly.

Jan <var data-var='date'>28</var>, <var data-var='time'>20:30</var> PST Completed - The scheduled maintenance has been completed.Jan <var data-var='date'>28</var>, <var data-var='time'>20:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Jan <var data-var='date'>28</var>, <var data-var='time'>14:37</var> PST Scheduled - We will be performing system maintenance at 8PM PST. APIs may return errors for approximately 2 minutes.

US production maintenance

Jan <var data-var='date'>13</var>, <var data-var='time'>14:50</var> PST Resolved - The SMS delivery issue with T-mobile has been resolved.Jan <var data-var='date'>13</var>, <var data-var='time'>14:15</var> PST Investigating - We are currently experiencing delivery issues with T-Mobile SMS 2FA messages. While AWS works to resolve the issue, SMS-based 2FA codes will be temporarily re-routed to email for affected users.

T-mobile SMS delivery issues in US

Sep <var data-var='date'>24</var>, <var data-var='time'>07:43</var> PDT Resolved - This incident has been resolved.Sep <var data-var='date'>24</var>, <var data-var='time'>07:34</var> PDT Update - We are continuing to monitor for any further issues.Sep <var data-var='date'>24</var>, <var data-var='time'>07:28</var> PDT Monitoring - A resolution has been rolled out - the team is monitoring system performance.Sep <var data-var='date'>24</var>, <var data-var='time'>07:07</var> PDT Identified - Root cause has been identified and we are actively working on a resolution.Sep <var data-var='date'>24</var>, <var data-var='time'>06:43</var> PDT Investigating - We are currently investigating reports of slow server response times and request timeouts

Keeper API website load time and API errors in US region

Jun <var data-var='date'>17</var>, <var data-var='time'>15:43</var> PDT Resolved - This issue has been resolved.Jun <var data-var='date'>17</var>, <var data-var='time'>14:26</var> PDT Update - We are continuing to work on a fix for this issue.Jun <var data-var='date'>17</var>, <var data-var='time'>12:51</var> PDT Identified - We have identified an issue with Keeper Secrets Manager Java SDK that is throwing API errors. A solution is underway and we'll be releasing an update on the Keeper backend API shortly. No changes are required on the SDK. This also affects Keeper Connection Manager integrations with KSM.

KSM Java SDK Errors (affecting Connection Manager)

Jun <var data-var='date'> 5</var>, <var data-var='time'>13:52</var> PDT Resolved - SMS delivery to T-mobile customers appears to be resolved.Jun <var data-var='date'> 5</var>, <var data-var='time'>10:14</var> PDT Investigating - SMS codes to T-mobile subscribers appears to be delayed or having delivery issues. Codes have been temporarily routed to email until the issue has been resolved. Please check your email for any 2FA codes.

Is KeeperSecurity down?

KeeperSecurity status is UP

Lastest published events

API errors during maintenance operation

Resolved: KeeperPAM Connection Errors in US Data Center

Resolved: KeeperPAM Connection Errors in US Data Center

KeeperSecurity Services, Components and Regions

Recent KeeperSecurity events

API errors during maintenance operation

Resolved: KeeperPAM Connection Errors in US Data Center

Resolved: KeeperPAM Connection Errors in US Data Center

Scheduled maintenance in AU data center

US GovCloud Region - Login Errors Resolved

US GovCloud region - maintenance planned tonight

Get your FREE account now,
5 monitors included!

Is KeeperSecurity down?

KeeperSecurity status is UP

Lastest published events

API errors during maintenance operation

Resolved: KeeperPAM Connection Errors in US Data Center

Resolved: KeeperPAM Connection Errors in US Data Center

KeeperSecurity Services, Components and Regions

Recent KeeperSecurity events

API errors during maintenance operation

Resolved: KeeperPAM Connection Errors in US Data Center

Resolved: KeeperPAM Connection Errors in US Data Center

Scheduled maintenance in AU data center

US GovCloud Region - Login Errors Resolved

US GovCloud region - maintenance planned tonight

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!