Jun <var data-var='date'>18</var>, <var data-var='time'>15:09</var> PDT Resolved - Send pipeline has resolved and enqueued messages are now sending properly. Jun <var data-var='date'>18</var>, <var data-var='time'>14:53</var> PDT Monitoring - Messages are being processed while our pipeline continues to process enqueued messages. Jun <var data-var='date'>18</var>, <var data-var='time'>14:31</var> PDT Update - The team has identified that the most recent deployment was not the cause of the regression. The team is still working on identifying the root cause. Messages will not be dropped, and they will be processed once the fix is out. Jun <var data-var='date'>18</var>, <var data-var='time'>13:35</var> PDT Update - The rollback deploy will land in about 30 minutes. Jun <var data-var='date'>18</var>, <var data-var='time'>13:11</var> PDT Identified - The team has identified a regression causing send requests to be stuck in a queued state. We have initiated a rollback to quickly resolve the issue. More updates to follow

Send Pipeline Latency

May <var data-var='date'>27</var>, <var data-var='time'>16:16</var> PDT Resolved - Please continue to monitor Slack's statuspage for any updates related to message deliverability: https://slack-status.com/ May <var data-var='date'>27</var>, <var data-var='time'>16:06</var> PDT Monitoring - Slack has reported messaging, and app degredation on their platform that could be impacting deliverability of Slack messages. This is a downstream provider issue and can be monitored on Slack's statuspage: https://slack-status.com/

Slack Deliverability Issues

Feb <var data-var='date'> 2</var>, <var data-var='time'>15:13</var> PST Resolved - All services operational. A public-facing RFO will be available in the coming days. Thank you for your understanding and continued partnership. Feb <var data-var='date'> 2</var>, <var data-var='time'>14:26</var> PST Update - EU Courier instance deployment has landed and services operational. Feb <var data-var='date'> 2</var>, <var data-var='time'>14:14</var> PST Update - EU instance redeployment is around 10 minutes out to land production. Feb <var data-var='date'> 2</var>, <var data-var='time'>14:01</var> PST Monitoring - The manual rollback has successfully landed, and all services are operational. The team is diligently monitoring and testing other services. Feb <var data-var='date'> 2</var>, <var data-var='time'>13:50</var> PST Update - The team has rolled back the deployment manually bypassing GH Actions outage. We're waiting for the deployment to land and confirm that all core functionality has been restored. Feb <var data-var='date'> 2</var>, <var data-var='time'>13:23</var> PST Update - The team is exploring alternate methods to redeploy the release until GH Actions recovers. Feb <var data-var='date'> 2</var>, <var data-var='time'>13:03</var> PST Update - The Courier team has a redeploy ready and waiting on GH Actions to recover from their degradation to publish the release. Feb <var data-var='date'> 2</var>, <var data-var='time'>12:44</var> PST Update - The current outage still impacts the current services: - webhooks - login - API endpoints - FE access. Our team is actively working on the redeployment. Feb <var data-var='date'> 2</var>, <var data-var='time'>12:29</var> PST Update - We are continuing to work on a fix for this issue. The following services are still impacted: - Login - Webapp Feb <var data-var='date'> 2</var>, <var data-var='time'>12:17</var> PST Update - Due to GH Actions experiencing an outage(https://www.githubstatus.com/), the redeploy took longer than expected and are now cutting a release. Feb <var data-var='date'> 2</var>, <var data-var='time'>12:07</var> PST Identified - The Courier team has encountered an issue on our platform impacting several services. The team has rolled out a redeploy to roll back changes.

Courier Multi Service Outage

Jan <var data-var='date'>29</var>, <var data-var='time'>02:00</var> PST Resolved - The Courier team identified an issue affecting observability metrics between 21:38 UTC and 22:13 UTC, during which metrics briefly experienced an outage. A fix was released and metrics are stabilized through observability channels. Metrics received during the outage window will not be accounted for in observability dashboards.

Observability Degradation

Jan <var data-var='date'>14</var>, <var data-var='time'>16:56</var> PST Resolved - Fix is out and application is stable Jan <var data-var='date'>14</var>, <var data-var='time'>13:47</var> PST Identified - The Courier team was alerted of a small degredation in the Courier web application. We have identified the issues and have rolled out a fix to mitigate. The current features will encounter some latency: - Audience Calculation - Audit Trail logging We will follow up with updates.

Courier Web App Performance

Oct <var data-var='date'>20</var>, <var data-var='time'>03:11</var> PDT Resolved - The incident has been resolved Oct <var data-var='date'>20</var>, <var data-var='time'>03:02</var> PDT Monitoring - Service has been restored. Engineering is working through system review and will continue to monitor. Oct <var data-var='date'>20</var>, <var data-var='time'>02:55</var> PDT Update - We are continuing to work on a fix for this issue. Oct <var data-var='date'>20</var>, <var data-var='time'>02:19</var> PDT Identified - An issue with our service provider, AWS, has impacted all Courier services. The engineering team is working on resolution.

All services impacted

Oct <var data-var='date'>16</var>, <var data-var='time'>23:01</var> PDT Resolved - Release is in production and Microsoft Teams messages are passing successfully. Oct <var data-var='date'>16</var>, <var data-var='time'>22:33</var> PDT Monitoring - The Courier team was alerted of an issue with MS Teams requests experiencing errors sending. We have identified the issue and published a fix that will land in production shortly

Microsoft Teams Tenant Id Errors

Oct <var data-var='date'>13</var>, <var data-var='time'>09:19</var> PDT Resolved - Courier experienced a slowdown in message delivery with Mailgun providers around 2AM PST. Our monitoring systems caught the spike in the bottleneck, which eventually stabilized. Our infrastructure team is actively monitoring.

Mailgun Message Delivery Latency

Oct <var data-var='date'> 8</var>, <var data-var='time'>02:00</var> PDT Resolved - Courier's delivery platform experienced significant latency around 1am PST. Our platform team managed to resolve the issue and is actively monitoring for any potential issues.

Message Delivery Latency

Jul <var data-var='date'> 7</var>, <var data-var='time'>14:28</var> PDT Resolved - Revert has landed and webapp is operational. Jul <var data-var='date'> 7</var>, <var data-var='time'>14:10</var> PDT Identified - An issue with logging in to the Courier platform and overall app performance was identified by our team and a revert of the offending release has been deployed. The estimated time for the release to land is 45 minutes.

Webapp HTTP Interaction Issues

Jun <var data-var='date'>12</var>, <var data-var='time'>13:36</var> PDT Resolved - WorkOS systems are back to normal. Jun <var data-var='date'>12</var>, <var data-var='time'>12:30</var> PDT Monitoring - Cloudflare has reported an outage that will impact SSO features provided by WorkOS. Customers with Okta SSO might experience some degradation when signing in. Our team is monitoring the status for both Cloudflare and WorkOS. WorkOS status: https://status.workos.com/incidents/k9s870cktcsf Cloudflare Status: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp

Cloudflare Outage Possible Okta SSO Impact

Jun <var data-var='date'> 5</var>, <var data-var='time'>13:08</var> PDT Resolved - Messages have recovered and pipeline is stable again. Jun <var data-var='date'> 5</var>, <var data-var='time'>12:19</var> PDT Update - Messages have stabilized but we will continue to monitor. Jun <var data-var='date'> 5</var>, <var data-var='time'>12:14</var> PDT Monitoring - Message bottleneck has cleared and the queue has stabilized. We will continue to monitor. Jun <var data-var='date'> 5</var>, <var data-var='time'>12:04</var> PDT Identified - Backed-up messages were caused by a bottleneck in our system, which has started to catch up. Messages will stabilize as the bottleneck clears. Jun <var data-var='date'> 5</var>, <var data-var='time'>11:51</var> PDT Investigating - The Courier team has identified an issue where there are significant delays of up to 10 minutes for messages. We are currently investigating and looking for a root cause to mitigate. Updates will follow.

Message Delays on Send Pipeline

May <var data-var='date'>16</var>, <var data-var='time'>14:02</var> PDT Resolved - The issue has been resolved. The root cause was purely a logging issue, notifications were still sent but showing up in logs as queued. All logs have been rehydrated May <var data-var='date'>16</var>, <var data-var='time'>07:53</var> PDT Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'>16</var>, <var data-var='time'>07:28</var> PDT Investigating - We are currently investigating an issue where EU workspace notifications appear to remain in the “queued” state.

EU Region – Notifications Stuck in “Queued” State

Apr <var data-var='date'>30</var>, <var data-var='time'>21:58</var> PDT Resolved - All impacted messages have been reprocessed. Apr <var data-var='date'>30</var>, <var data-var='time'>18:06</var> PDT Update - The team is still working through the reprocessing for impacted messages. Confirmation of all impacted messages reprocessed will be updated here. Apr <var data-var='date'>30</var>, <var data-var='time'>16:26</var> PDT Monitoring - Fix has landed and the team is monitoring progress. Messages sent to AWS SES are sending normally. The team will reprocess impacted messages. Apr <var data-var='date'>30</var>, <var data-var='time'>15:47</var> PDT Update - The reverted changes will land in production in around one hour. The team will investigate the impact and handle message retries accordingly. Apr <var data-var='date'>30</var>, <var data-var='time'>15:39</var> PDT Identified - The Courier team has identified an issue with a recent deployment that is impacting AWS SES integrations. We have found the root cause and have reverted the changes. The team is gathering impact.

AWS SES Failures

Feb <var data-var='date'>28</var>, <var data-var='time'>20:11</var> PST Resolved - All impacted messages have been reprocessed. Feb <var data-var='date'>28</var>, <var data-var='time'>11:52</var> PST Update - The team is still working on reprocessing dropped messages caused by this issue. Send pipeline is operational. Feb <var data-var='date'>28</var>, <var data-var='time'>10:57</var> PST Update - Messages with tags are no longer impacted, and error has been resolved. The team is working on reprocessing dropped messages. Feb <var data-var='date'>28</var>, <var data-var='time'>10:40</var> PST Monitoring - The fix has landed in production and we will be monitoring for any more "Internal Courier Errors". We will be working on reprocessing any dropped messages caused by this issue. Feb <var data-var='date'>28</var>, <var data-var='time'>09:36</var> PST Update - A fix has been merged and will fully deploy within the next hour. Updates to follow. Feb <var data-var='date'>28</var>, <var data-var='time'>09:14</var> PST Update - We have tested the fix for "Internal Courier Error" and will be releasing it to production soon. Feb <var data-var='date'>28</var>, <var data-var='time'>08:48</var> PST Identified - The Courier team has identified an issue where messages fail due to an "Internal Courier Error" in the test environment. Our team found the root cause and are working to mitigate the issue.

Internal Courier Error Issue

Feb <var data-var='date'>21</var>, <var data-var='time'>09:10</var> PST Resolved - Segment event processing stream has resolved. Feb <var data-var='date'>21</var>, <var data-var='time'>07:55</var> PST Monitoring - The Courier team has identified a slowdown in Segment event processing. There will be a slight delay for messages triggered by incoming Segment triggers. We estimate the queue to normalize soon and are monitoring.

Segment Event Processing Slowdowns

Feb <var data-var='date'>18</var>, <var data-var='time'>06:47</var> PST Resolved - Message sends are flowing normally, and events are caught up without delays. Feb <var data-var='date'>18</var>, <var data-var='time'>06:29</var> PST Monitoring - We're monitoring our processing pipeline as messages are flowing normally. There is a slight delay in event processing which will also impact outbound webhooks. Feb <var data-var='date'>18</var>, <var data-var='time'>06:22</var> PST Identified - The Courier team identified a delay in message processing and event updating. The pipeline queue for message delivery has normalized, however significant slowdowns in message events are still processing.

Significant Notification Event Latency

Feb <var data-var='date'>13</var>, <var data-var='time'>15:09</var> PST Resolved - The team has identified the issue related to network connections on AWS which resulted in a backup of messages on our pipeline. Messages are flowing through as expected, and the stream has caught up. Feb <var data-var='date'>13</var>, <var data-var='time'>14:51</var> PST Monitoring - Our team has identified a bottleneck in our system caused by timeouts on our sendworker. Our exponential backoff has caught up and messages are starting to go through. Feb <var data-var='date'>13</var>, <var data-var='time'>14:32</var> PST Identified - The Courier team has identified a slowdown in sending notifications. This issue has been identified by our team and is closely monitoring. We do not expect any messages to drop. We will continue to monitor and update accordingly.

Courier Send and Message Event Slowdown

Feb <var data-var='date'>11</var>, <var data-var='time'>15:08</var> PST Resolved - The send stream has caught up, all messages have been sent and the issue appears to be fully resolved. Feb <var data-var='date'>11</var>, <var data-var='time'>14:34</var> PST Monitoring - The team has identified the cause of message sends and slowdowns to be related to an AWS service. Messages that were stuck in a queued state are slowly passing through. The team will continue to monitor and update. Feb <var data-var='date'>11</var>, <var data-var='time'>14:15</var> PST Identified - The Courier team has identified a slowdown in our pipeline. The queue has backed up but no messages have been dropped at the moment. We will continue to update as we monitor our pipeline.

Delay in Message Sends and Processing

Jan <var data-var='date'>10</var>, <var data-var='time'>20:25</var> PST Resolved - The service issue affecting automation workflows with delay steps has been partially resolved. New automation workflows started after 6:05 PM PT are executing successfully. However, we have determined that automations that failed during the incident period (approximately 3:20 PM PT - 6:05 PM PT) cannot be automatically retried at this time due to technical limitations. If your business was impacted by failed automation runs during this incident, please contact our support team and we will work directly with you to address your concerns. We sincerely apologize for this disruption to your workflows. Jan <var data-var='date'>10</var>, <var data-var='time'>18:19</var> PST Monitoring - We have deployed a fix for the earlier issue affecting automation workflows with delay steps. New automation workflows are now executing successfully. However, automations that failed during the incident period (approximately 3:20 PM PT - 6:05 PM PT) have not yet been automatically retried. Our engineering team is monitoring the recovery and evaluating a plan to process these backlogged automations. Jan <var data-var='date'>10</var>, <var data-var='time'>17:23</var> PST Identified - We are currently experiencing an issue affecting automation workflows that include delay steps. Some customers may encounter failures when attempting to execute automations with scheduled delays. Our engineering team has identified the root cause and is implementing a fix. We expect the service to be fully restored in a short while.

Automation Delay Processing Issues

Oct <var data-var='date'>24</var>, <var data-var='time'>19:48</var> PDT Resolved - This incident has been resolved Oct <var data-var='date'>24</var>, <var data-var='time'>19:43</var> PDT Monitoring - A fix has been implemented and delivery times are beginning to returning to expected levels. The team will continue to monitor. Oct <var data-var='date'>24</var>, <var data-var='time'>18:31</var> PDT Update - The team identified an issue with the latest fix and reverted it. A new fix has been published to mitigate the message delay issue. ETA to land in production ~45 minutes. Oct <var data-var='date'>24</var>, <var data-var='time'>17:40</var> PDT Update - Fix has been deployed and will be live in around 1hr. Oct <var data-var='date'>24</var>, <var data-var='time'>17:00</var> PDT Identified - The team has identified the issue and is rolling out a fix for the message delays. Oct <var data-var='date'>24</var>, <var data-var='time'>16:29</var> PDT Update - The team is still investigating and discussing the root cause internally. Messages are experiencing longer than normal processing times. Oct <var data-var='date'>24</var>, <var data-var='time'>16:01</var> PDT Update - We are continuing to monitor the root cause of the delayed message processing. Oct <var data-var='date'>24</var>, <var data-var='time'>15:38</var> PDT Investigating - Courier has identified a delay in message processing for the messages API. We are currently investigating the root cause and will update you periodically.

Delayed Message Processing

Oct <var data-var='date'>21</var>, <var data-var='time'>11:51</var> PDT Resolved - The data stream is unblocked, and the message logs queue is resolved and flowing normally. Oct <var data-var='date'>21</var>, <var data-var='time'>11:30</var> PDT Monitoring - The fix has landed in production and the team is monitoring the message log queue. Message event logs should be flowing normally. Oct <var data-var='date'>21</var>, <var data-var='time'>10:19</var> PDT Update - The team encountered an issue with testing the fix and reverted the update. We are publishing a new update that should resolve the backlogged message logs. Oct <var data-var='date'>21</var>, <var data-var='time'>09:07</var> PDT Identified - The team is testing out a fix to reduce the bottlenecked log lines before releasing to production. Oct <var data-var='date'>21</var>, <var data-var='time'>08:56</var> PDT Update - We are continuing to investigate this issue. Oct <var data-var='date'>21</var>, <var data-var='time'>08:56</var> PDT Investigating - The Courier team is investigating an issue with the event logger for message event logs hitting a bottleneck. The team is actively investigating. Messages are still sending.

Message Logs Delays

Oct <var data-var='date'>10</var>, <var data-var='time'>19:33</var> PDT Resolved - Inbox connectivity issues resolved. Oct <var data-var='date'>10</var>, <var data-var='time'>13:51</var> PDT Update - The team has initiated backlogging the impacted Inbox messages. Oct <var data-var='date'>10</var>, <var data-var='time'>12:07</var> PDT Update - The team has gathered impacted messages and is running tests before a release. Oct <var data-var='date'>10</var>, <var data-var='time'>11:06</var> PDT Update - We are continuing to monitor for any further issues. Oct <var data-var='date'>10</var>, <var data-var='time'>10:50</var> PDT Update - We are continuing to gather a list of impacted messages to backfill impacted Inbox components. Oct <var data-var='date'>10</var>, <var data-var='time'>09:46</var> PDT Monitoring - Inbox connectivity for fetching messages has been reestablished. The team is monitoring closely, and working on retroactively processing impacted messages to Inbox Oct <var data-var='date'>10</var>, <var data-var='time'>09:30</var> PDT Identified - The team has identified the issue and will be rolling out a fix. The team will reprocess the impacted messages that were not fetched. Oct <var data-var='date'>10</var>, <var data-var='time'>08:38</var> PDT Investigating - The Courier Team is investigating an issue related to Inbox fetching messages in the component.

Courier Inbox FetchMessages Connectivity Issues

Sep <var data-var='date'> 6</var>, <var data-var='time'>09:59</var> PDT Resolved - Bottleneck has cleared for messages and event statuses. Sep <var data-var='date'> 6</var>, <var data-var='time'>09:52</var> PDT Update - Corrective actions have cleared the bottleneck and messages and events should be flowing normally. Sep <var data-var='date'> 6</var>, <var data-var='time'>09:44</var> PDT Monitoring - The team has increased the send pipeline worker and messages are clearing the bottleneck. Sep <var data-var='date'> 6</var>, <var data-var='time'>09:39</var> PDT Update - The team has increased our processing to help with the bottleneck. Sep <var data-var='date'> 6</var>, <var data-var='time'>09:35</var> PDT Identified - The Courier team has identified an issue with the send pipeline and event status updates causing queued messages and delayed webhook events. The Courier team will monitor the bottleneck and adjust message batching necessary to flow messages normally.

Send Pipeline and Event Status Slowdowns

Aug <var data-var='date'>19</var>, <var data-var='time'>14:35</var> PDT Resolved - The general pipeline has recovered. Aug <var data-var='date'>19</var>, <var data-var='time'>13:02</var> PDT Update - Fix has been deployed, and enqueued messages have started to go through slowly. Once the bottleneck clears, messages should start to flow normally. Aug <var data-var='date'>19</var>, <var data-var='time'>11:56</var> PDT Update - Release is published and building to production. ETA ~45 minutes. Aug <var data-var='date'>19</var>, <var data-var='time'>11:50</var> PDT Update - The release is live, and the team is monitoring it. Aug <var data-var='date'>19</var>, <var data-var='time'>11:48</var> PDT Monitoring - Our team has released a revert to address the regression and it's in the process of merging. Aug <var data-var='date'>19</var>, <var data-var='time'>11:32</var> PDT Identified - The Courier team identified an issue in our health monitoring involving our message event processing. The issue has been identified and a revert is in place.

Is Courier down?

Courier status is UP

Lastest published events

Send Pipeline Latency

Slack Deliverability Issues

Courier Multi Service Outage

Courier Services, Components and Regions

Recent Courier events

Send Pipeline Latency

Slack Deliverability Issues

Courier Multi Service Outage

Observability Degradation

Courier Web App Performance

All services impacted