Aug <var data-var='date'>18</var>, <var data-var='time'>20:00</var> EDT Resolved - On August 18 from 23:59 to 00:03 UTC (August 19), a subset of API requests (primarily AuthKit authentication endpoints) returned HTTP 500 errors. Automatic maintenance of a large, seldom-used internal index caused brief write-ahead-log contention on a primary database that exhausted connection capacity; service recovered automatically within four minutes. We identified the root cause and removed the index from production at 03:53 UTC on August 19 to prevent recurrence.

Elevated errors on AuthKit authentication endpoints

Aug <var data-var='date'>16</var>, <var data-var='time'>18:04</var> EDT Resolved - Microsoft's Outlook team has confirmed that mitigations have been implemented. We've verified that email delivery to Microsoft 365 domains is operating normally again. This incident has been resolved. Aug <var data-var='date'>15</var>, <var data-var='time'>12:21</var> EDT Identified - We continue to see email deliverability issues to a subset of Microsoft 365 tenant domains. This does not affect consumer domains such as outlook.com, hotmail.com, live.com or msn.com. We will provide updates as they become available. Aug <var data-var='date'>14</var>, <var data-var='time'>23:32</var> EDT Monitoring - We are observing improvements in deliverability to Microsoft email addresses, in line with the resolution of an incident with our upstream email service provider. We will continue to monitor. Aug <var data-var='date'>14</var>, <var data-var='time'>21:35</var> EDT Identified - We continue to experience deliverability delays to a subset of users with email addresses hosted by Microsoft / Office 365. We are working with our email providers to mitigate the issues, we will provide updates as they become available. Customers using their own custom email providers are not affected. Aug <var data-var='date'>14</var>, <var data-var='time'>20:24</var> EDT Investigating - We are seeing deliverability issues with emails sent to Microsoft / Outlook recipients.

Delayed Email Deliverability

Aug <var data-var='date'>14</var>, <var data-var='time'>16:45</var> EDT Resolved - This incident has been resolved. Email delivery was delayed between 15:45 EDT and 16:35 EDT due to an issue with an upstream email service provider. This affected customers who have not yet configured a custom email domain or a custom email provider. Aug <var data-var='date'>14</var>, <var data-var='time'>16:44</var> EDT Monitoring - We've mitigated the impact and email delivery is returning to normal. Aug <var data-var='date'>14</var>, <var data-var='time'>16:14</var> EDT Investigating - We're investigating delays in email deliverability.

Aug <var data-var='date'>11</var>, <var data-var='time'>15:24</var> EDT Resolved - This incident has been resolved. Aug <var data-var='date'>11</var>, <var data-var='time'>14:02</var> EDT Investigating - We are investigating intermittent sign-in failures on the WorkOS Dashboard. We will share an update as soon as possible.

Intermittent Dashboard sign-in errors

Aug <var data-var='date'> 5</var>, <var data-var='time'>14:28</var> EDT Resolved - This was an issue with our internal telemetry - there is no customer impact. Sorry for the false alarm. Aug <var data-var='date'> 5</var>, <var data-var='time'>14:04</var> EDT Investigating - We are investigating an issue with our webhooks. We apologize for the inconvenience and will share an update once we have more information.

Webhook Telemetry Issue

Jul <var data-var='date'>31</var>, <var data-var='time'>14:06</var> EDT Resolved - The issue has been resolved. Jul <var data-var='date'>31</var>, <var data-var='time'>13:10</var> EDT Monitoring - The fix has deployed, we are monitoring systems to verify the impact is fully mitigated Jul <var data-var='date'>31</var>, <var data-var='time'>13:00</var> EDT Update - We have identified the issue and are deploying a fix. AuthKit SSO sign-on remains partially degraded for a small subset of customers; standalone SSO continues to be unaffected. We'll share another update once the fix has fully rolled out. Jul <var data-var='date'>31</var>, <var data-var='time'>12:34</var> EDT Update - The issue affects a small set of customers using Authkit SSO. The standalone SSO API remains unaffected, we have corrected that here on the status page Jul <var data-var='date'>31</var>, <var data-var='time'>12:16</var> EDT Update - We have identifed an issue causing a small subset of Authkit SSO customers to experience errors during sign-on. A fix is being prepared. Jul <var data-var='date'>31</var>, <var data-var='time'>12:12</var> EDT Update - It appears some users may be experiencing a redirect loop during sign-on, we are narrowing in on a cause Jul <var data-var='date'>31</var>, <var data-var='time'>12:01</var> EDT Investigating - We're investigating an increase in Single Sign-On errors. We will share an update as soon as possible.

Elevated Authkit SSO Errors

Jul <var data-var='date'>29</var>, <var data-var='time'>15:54</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'>29</var>, <var data-var='time'>15:24</var> EDT Monitoring - We have remediated the issue. Email delivery rates have recovered and reported email delivery status in Dashboard are being updated properly. We are continuing to monitor. Jul <var data-var='date'>29</var>, <var data-var='time'>14:36</var> EDT Identified - We've identified the issue and have implemented a mitigation. We are seeing normal email deliverability now but Dashboard email delivery statuses are currently not being updated. Jul <var data-var='date'>29</var>, <var data-var='time'>14:30</var> EDT Investigating - We're investigating delays in email deliverability.

Jul <var data-var='date'>27</var>, <var data-var='time'>17:12</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'>27</var>, <var data-var='time'>17:11</var> EDT Update - This has been resolved. Jul <var data-var='date'>27</var>, <var data-var='time'>16:54</var> EDT Monitoring - We have confirmed that webhook delivery times are back to normal levels. We are continuing to monitor. Jul <var data-var='date'>27</var>, <var data-var='time'>16:34</var> EDT Update - We have rolled out a fix and are seeing improvements to webhook delivery times. We will provide another update soon. Jul <var data-var='date'>27</var>, <var data-var='time'>15:36</var> EDT Identified - We have identified the cause of the delays in webhook delivery and are rolling out a fix to reduce processing latency. Jul <var data-var='date'>27</var>, <var data-var='time'>15:09</var> EDT Investigating - We're investigating delays in webhook delivery. We will share an update as soon as possible.

Webhook Delivery Delays

Jul <var data-var='date'>27</var>, <var data-var='time'>13:37</var> EDT Resolved - Email delivery is operating normally. Jul <var data-var='date'>27</var>, <var data-var='time'>11:36</var> EDT Monitoring - We have remediated the issue. Email delivery rates have recovered and reported email delivery status in Dashboard are being updated properly. We are continuing to monitor. Jul <var data-var='date'>27</var>, <var data-var='time'>10:33</var> EDT Update - We are continuing to investigate email delivery errors in addition to Dashboard email delivery statuses not being updated. Jul <var data-var='date'>27</var>, <var data-var='time'>10:10</var> EDT Investigating - We are investigating reports that email delivery statuses in Dashboard are not being updated.

Continued email delivery errors

Jul <var data-var='date'>27</var>, <var data-var='time'>09:04</var> EDT Resolved - Delivery rates have recovered. Jul <var data-var='date'>27</var>, <var data-var='time'>08:11</var> EDT Monitoring - We have applied a refined mitigation and are continuing to monitor delivery rates. Jul <var data-var='date'>27</var>, <var data-var='time'>07:20</var> EDT Investigating - Our mitigation did not fully resolve the issue. We are continuing to investigate. Jul <var data-var='date'>27</var>, <var data-var='time'>07:03</var> EDT Monitoring - We have implemented a mitigation and are continuing to monitor. Jul <var data-var='date'>27</var>, <var data-var='time'>06:05</var> EDT Investigating - We are investigating elevated errors during delivery of emails.

Elevated errors during email delivery

Jul <var data-var='date'>21</var>, <var data-var='time'>22:51</var> EDT Resolved - This has been resolved. Jul <var data-var='date'>21</var>, <var data-var='time'>22:19</var> EDT Monitoring - We've remediated the issue and are seeing email delivery recover. Jul <var data-var='date'>21</var>, <var data-var='time'>22:16</var> EDT Identified - We've identified the issue and are implementing remediations. Jul <var data-var='date'>21</var>, <var data-var='time'>21:35</var> EDT Investigating - We're investigating possible email delivery delays.

Email Delivery Delays

Jul <var data-var='date'>21</var>, <var data-var='time'>17:54</var> EDT Resolved - We identified and fixed the issue. AuthKit SSO has fully recovered. Jul <var data-var='date'>21</var>, <var data-var='time'>17:21</var> EDT Investigating - We are investigating SSO failures for AuthKit customers.

AuthKit SSO failures

Jul <var data-var='date'>21</var>, <var data-var='time'>15:49</var> EDT Resolved - Webhook latency is back to normal levels. We have identified that this latency only affected a very small subset of customers. Jul <var data-var='date'>21</var>, <var data-var='time'>15:37</var> EDT Monitoring - We're seeing a return to normal webhook latency and are continuing to monitor. Jul <var data-var='date'>21</var>, <var data-var='time'>15:09</var> EDT Identified - We've identified the source of the latency and have implemented a fix. Jul <var data-var='date'>21</var>, <var data-var='time'>14:58</var> EDT Investigating - We're investigating webhook delivery delays. We apologize for the inconvenience and will share an update once we have more information.

Elevated Webhook Latency

Jul <var data-var='date'>16</var>, <var data-var='time'>08:21</var> EDT Resolved - We've resolved the issue with email delivery delays. Jul <var data-var='date'>16</var>, <var data-var='time'>08:18</var> EDT Identified - We've identified the issue causing email delays. We are working to mitigate the issue. Jul <var data-var='date'>16</var>, <var data-var='time'>08:06</var> EDT Investigating - We're seeing delays in email delivery. Our team is investigating. We will share an update as soon as possible.

Email delivery delays

Jul <var data-var='date'>16</var>, <var data-var='time'>06:44</var> EDT Resolved - This incident is resolved. We experienced elevated error rates and latencies between 8am and 10:10am UTC. We will deliver a full postmortem as soon as possible. Jul <var data-var='date'>16</var>, <var data-var='time'>06:30</var> EDT Monitoring - We have deployed the fix and we are now monitoring our systems to ensure the issue is resolved. Jul <var data-var='date'>16</var>, <var data-var='time'>06:11</var> EDT Identified - We've identified the cause of the elevated platform errors. We've implemented a fix and are seeing error rates decrease. Jul <var data-var='date'>16</var>, <var data-var='time'>05:58</var> EDT Update - We continue to investigate elevated error rates across our platform. Jul <var data-var='date'>16</var>, <var data-var='time'>05:25</var> EDT Update - We continue to investigate elevated error rates across our platform. We will share an update as soon as possible. Jul <var data-var='date'>16</var>, <var data-var='time'>04:26</var> EDT Investigating - We are seeing elevated error rates across our platform. Our team is investigating. We will share an update as soon as possible.

Elevated Platform Errors

Jul <var data-var='date'>10</var>, <var data-var='time'>16:36</var> EDT Resolved - We have restored full service to the standalone MFA API endpoints. Jul <var data-var='date'>10</var>, <var data-var='time'>15:26</var> EDT Monitoring - Beginning at at 16:00 UTC, users of our standalone MFA API experienced intermittent elevated error rates for approximately 2 hours. MFA through AuthKit was not affected. This issue has been resolved, and we are continuing to monitor.

Elevated error rates for standalone MFA requests

Jun <var data-var='date'>25</var>, <var data-var='time'>01:12</var> EDT Resolved - The remaining out-of-sync user records are now back in sync. Jun <var data-var='date'>24</var>, <var data-var='time'>23:29</var> EDT Update - Synchronization has continued to catch up. We are continuing to monitor the few out of sync changes that are remaining. Jun <var data-var='date'>24</var>, <var data-var='time'>20:09</var> EDT Monitoring - Provisioning has caught up for user creation, and new directory activity is syncing normally. We are still working through and monitoring the remaining synchronization catch-up, which is not yet fully complete. Another update to follow. Jun <var data-var='date'>24</var>, <var data-var='time'>13:37</var> EDT Update - We have corrected the affected directories, and they are back to a provisioning status. Jun <var data-var='date'>24</var>, <var data-var='time'>12:47</var> EDT Identified - The underlying issue that has lead to a delay in provisioning has been identified, and a fix is being prepared Jun <var data-var='date'>22</var>, <var data-var='time'>17:36</var> EDT Investigating - We are seeing a delay in Directory Provisioning for Authkit, for any directories created before June 22, 21:36:17 UTC. Any directories created after this time are working as expected.

Degraded performance for Directory Provisioning for Authkit

Jun <var data-var='date'>22</var>, <var data-var='time'>14:01</var> EDT Resolved - From 13:35 UTC to 14:24 UTC users connecting through our Chicago (ORD) edge node may have experienced elevated API error rates. We have resolved the issue and are monitoring. Jun <var data-var='date'>22</var>, <var data-var='time'>10:42</var> EDT Monitoring - We detected elevated errors from our Chicago (ORD) edge node. Errors have since recovered and we're continuing to monitor. Jun <var data-var='date'>22</var>, <var data-var='time'>10:03</var> EDT Investigating - We are investigating elevated error rates across our API. We apologize for the inconvenience and will share an update once we have more information.

Elevated API Errors

Jun <var data-var='date'> 9</var>, <var data-var='time'>05:00</var> EDT Resolved - From 9:43 to 10:02 UTC SSO endpoints experienced increased latency and intermittent timeouts. We have resolved the issue and are monitoring.

SSO Request Latency

Jun <var data-var='date'> 4</var>, <var data-var='time'>20:49</var> EDT Resolved - The backlog of messages have been sent and system performance has returned to its expected state. Sending delays occurred between 2:50pm – 5:18pm PT. Jun <var data-var='date'> 4</var>, <var data-var='time'>20:34</var> EDT Monitoring - Email delivery has recovered. We're continuing to monitor. Jun <var data-var='date'> 4</var>, <var data-var='time'>20:08</var> EDT Update - Our email provider is continuing to work through the backlog and has scaled up to speed up the process. Jun <var data-var='date'> 4</var>, <var data-var='time'>19:22</var> EDT Identified - Email delivery for customers without custom domains is delayed due to an issue with our upstream service provider.

May <var data-var='date'>28</var>, <var data-var='time'>22:46</var> EDT Resolved - The issue affecting the Events API GET /events endpoint has been resolved. The fix has been deployed, and affected requests are now processing normally. May <var data-var='date'>28</var>, <var data-var='time'>22:30</var> EDT Identified - We have identified the cause of the elevated errors affecting the GET /events API and are working on a resolution. We’ll share another update soon.

Elevated Events API Errors

May <var data-var='date'>22</var>, <var data-var='time'>14:58</var> EDT Resolved - This incident has been resolved. May <var data-var='date'>22</var>, <var data-var='time'>14:55</var> EDT Monitoring - A fix has been deployed for the issue affecting static asset loading on the WorkOS Dashboard, Admin Portal, Docs, and hosted result pages. We are continuing to monitor for full recovery. May <var data-var='date'>22</var>, <var data-var='time'>14:32</var> EDT Update - We're aware of a styling issue currently affecting the WorkOS Dashboard. CSS assets are failing to load, leaving the Dashboard in an unstyled and difficult-to-use state. Underlying APIs, authentication, and data are not impacted - this is limited to the Dashboard UI. We're investigating and will update shortly. May <var data-var='date'>22</var>, <var data-var='time'>14:29</var> EDT Investigating - We are investigating an issue with our Dashboard. We will share an update as soon as possible.

Dashboard and Docs unavailable

May <var data-var='date'>19</var>, <var data-var='time'>18:00</var> EDT Resolved - The webhooks queue backlog has been delivered. Thank you for your patience. May <var data-var='date'>19</var>, <var data-var='time'>17:43</var> EDT Identified - We are experiencing delays delivering a subset of webhooks. We have identified the issue and working on a remediation.

Webhook delivery delays

May <var data-var='date'>14</var>, <var data-var='time'>11:12</var> EDT Resolved - This issue is now resolved. May <var data-var='date'>14</var>, <var data-var='time'>11:10</var> EDT Monitoring - We experienced errors in our OIDC connections. The issue is now resolved.

OIDC Errors

May <var data-var='date'>12</var>, <var data-var='time'>16:29</var> EDT Resolved - Newly-sent emails from WorkOS are now displaying properly in Gmail again. May <var data-var='date'>12</var>, <var data-var='time'>15:43</var> EDT Identified - We've identified the issue and are applying a fix. May <var data-var='date'>12</var>, <var data-var='time'>15:37</var> EDT Update - We are investigating reports of AuthKit emails rendering with no copy in some email clients. May <var data-var='date'>12</var>, <var data-var='time'>15:14</var> EDT Investigating - We are investigating reports of some AuthKit emails, such as invitations, being delivered with empty copy.

Is WorkOS down?

WorkOS status is UP

Lastest published events

Elevated errors on AuthKit authentication endpoints

Delayed Email Deliverability

Delayed Email Deliverability

WorkOS Services, Components and Regions

Recent WorkOS events

Elevated errors on AuthKit authentication endpoints

Delayed Email Deliverability

Delayed Email Deliverability

Intermittent Dashboard sign-in errors

Webhook Telemetry Issue

Elevated Authkit SSO Errors