Mar <var data-var='date'>23</var>, <var data-var='time'>21:16</var> UTC Resolved - This incident has been resolved. We're still collecting more information on the root cause here; we'll update this status post when we have more info.Mar <var data-var='date'>23</var>, <var data-var='time'>20:31</var> UTC Monitoring - It seems that SignalWire has resolved their issue. We're monitoring for any other problems.Mar <var data-var='date'>23</var>, <var data-var='time'>20:29</var> UTC Update - SignalWire has identified an issue causing problems with incoming and outgoing PSTN/SIP calls for Daily customers. If you're making dialout calls, you may see an error that says "DialOut stopped: Remote busy or Remote did not answer or Remote ended the call". If you're using dialin, you may hear a message that says "the number you have dialed is not configured correctly and cannot receive calls" or something similar.Mar <var data-var='date'>23</var>, <var data-var='time'>20:18</var> UTC Investigating - We're investigating elevated error rates for incoming PSTN/SIP calls.

Issues with inbound phone calls

Mar <var data-var='date'>16</var>, <var data-var='time'>21:57</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>16</var>, <var data-var='time'>18:31</var> UTC Monitoring - We've identified an issue with the webhook system that was causing brief delays when finding meeting start and end events for webhooks. We've addressed that issue. Webhook delivery time was only minimally impacted during the event, and it has returned to normal as of about 15 minutes ago.Mar <var data-var='date'>16</var>, <var data-var='time'>17:12</var> UTC Investigating - We're investigating an issue that may be causing delayed delivery for some webhooks.

Webhook delivery delays

Feb <var data-var='date'>18</var>, <var data-var='time'>15:55</var> UTC Resolved - This incident has been resolved.Feb <var data-var='date'>18</var>, <var data-var='time'>14:19</var> UTC Monitoring - The fix from our upstream carrier partner has been applied, and service has been restored for SIP/PSTN dial-in (inbound) calling. We are seeing successful inbound call connections again, and our internal testing confirms recovery. We are continuing to monitor closely to ensure stability and full recovery.Feb <var data-var='date'>18</var>, <var data-var='time'>14:18</var> UTC Identified - Our upstream carrier partner has identified the root cause of the issue impacting SIP/PSTN dial-in (inbound) calling and has implemented a fix. Service is beginning to recover, though some degradation may still be present while the fix fully propagates. We are actively monitoring and working to confirm full restoration of inbound calling. Outbound (dial-out) calling remains unaffected.Feb <var data-var='date'>18</var>, <var data-var='time'>11:31</var> UTC Investigating - We are currently investigating an issue affecting SIP / PSTN inbound calls. Customers may encounter 480 “Temporarily Unavailable” errors when attempting to receive calls. Our engineering team is actively working to identify the root cause and restore service.

SIP / PSTN Disruption

Feb <var data-var='date'> 5</var>, <var data-var='time'>08:15</var> UTC Resolved - The issue has been resolved. We’ve seen the number of “Failed to load call object bundle” errors return to normal levels, and call joins are operating as expected.Feb <var data-var='date'> 5</var>, <var data-var='time'>01:44</var> UTC Monitoring - The initial increase in error rates has subsided. Based on reports from a few customers, it seems like the impact was primarily in the Texas area. We're continuing to work with AWS to determine the root cause.Feb <var data-var='date'> 4</var>, <var data-var='time'>23:59</var> UTC Identified - We've seen an increase in "Failed to load call object bundle" errors within the past hour. This happens when the browser is unable to download our JavaScript bundle from the CDN. If this happens to you or your users, you should be able to refresh the page and join your call successfully. We're working with our CDN provider to identify the issue.

Increase in "Failed to load call object bundle" errors

Feb <var data-var='date'> 2</var>, <var data-var='time'>22:37</var> UTC Resolved - This incident has been resolved.Feb <var data-var='date'> 2</var>, <var data-var='time'>18:34</var> UTC Monitoring - We're seeing extremely low error rates, but Deepgram hasn't resolved their incident yet, so we'll keep monitoring.Feb <var data-var='date'> 2</var>, <var data-var='time'>16:56</var> UTC Update - We're continuing to see very brief occurrences of slow or missing transcription data from the Deepgram streaming endpoint. We're talking to Deepgram engineers and continuing to monitor the situation.Feb <var data-var='date'> 2</var>, <var data-var='time'>15:45</var> UTC Identified - We're tracking a Deepgram incident that is causing problems with transcription events, as well as captions in Daily Prebuilt: https://status.deepgram.com/incidents/rp0jzltym9kw

Issues with transcriptions and captions

Jan <var data-var='date'>26</var>, <var data-var='time'>21:24</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'>26</var>, <var data-var='time'>20:53</var> UTC Monitoring - We've worked through the webhook backlog, and delivery times should be normal. We'll be monitoring for any further issues.Jan <var data-var='date'>26</var>, <var data-var='time'>19:45</var> UTC Update - We're continuing to work through a backlog of webhook events, and the delay is starting to decrease. It will still take some time to get fully caught up.Jan <var data-var='date'>26</var>, <var data-var='time'>18:03</var> UTC Identified - We’ve identified the cause of the issue. While a backlog of webhooks remains, we’re actively working to reduce it. Webhook deliveries will continue to be delayed for some time, but no data is being lost.Jan <var data-var='date'>26</var>, <var data-var='time'>16:47</var> UTC Investigating - We are currently experiencing an abnormally large queue of webhooks awaiting delivery. Our team is investigating and working to restore normal processing.

Webhook deliveries delayed

Jan <var data-var='date'>15</var>, <var data-var='time'>22:11</var> UTC Resolved - This incident has been resolved.Jan <var data-var='date'>15</var>, <var data-var='time'>21:35</var> UTC Update - We've rolled back the release that was in process when we started seeing the errors, which seems to have resolved the issue. We'll be monitoring for any regressions.Jan <var data-var='date'>15</var>, <var data-var='time'>21:32</var> UTC Identified - We're seeing an increased rate of errors when dialing out SIP/PSTN calls.

Issues starting SIP/PSTN calls

Jan <var data-var='date'>13</var>, <var data-var='time'>23:46</var> UTC Resolved - Webhook delivery times have returned to normal.Jan <var data-var='date'>13</var>, <var data-var='time'>23:00</var> UTC Update - We're continuing to work through the backlog of webhooks.Jan <var data-var='date'>13</var>, <var data-var='time'>22:13</var> UTC Identified - We're seeing an abnormally large queue of webhooks to be delivered, so we've scaled up to work through the backlog. We'll post another update when delivery times return to normal.

Jan <var data-var='date'>13</var>, <var data-var='time'>00:16</var> UTC Resolved - SignalWire experienced an outage earlier this morning. Their status site post is here: https://status.signalwire.com/incidents/MJVYwKKYYwbG Our first customer report of problems was at 1:51 AM PST (09:51 UTC). During the incident, many SignalWire SIP calls were returning 603 Declined. As we were investigating, SignalWire opened an incident on their status site at 2:26 AM PST (10:26 UTC). Their incident post didn't provide enough information for us to determine the overall impact to our customers. As we continued investigating our own logs and pressing SignalWire for more information, our affected customers started reporting that the issue was resolved around 2:35 AM PST. SignalWire ultimately resolved their incident at 03:27 PST.

Issues with PSTN/SIP availability

Oct <var data-var='date'>21</var>, <var data-var='time'>15:22</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>21</var>, <var data-var='time'>00:50</var> UTC Monitoring - AWS has resolved their incident. We're continuing to run the same call servers in us-east-1 as we were before the incident started, but we're still routing the majority of US traffic through us-west-2. We plan to return things to normal tomorrow.Oct <var data-var='date'>20</var>, <var data-var='time'>15:10</var> UTC Update - AWS is continuing to address issues in us-east-1. We're still seeing normal activity levels across our APIs and call servers for this time of day, so we have no reason to believe that any your customers' calls are actually being affected. Part of AWS's mitigation efforts involve limiting the rate at which we can start new call servers. Out of an abundance of caution, we're temporarily removing us-east-1 from our list of available regions for new call sessions. This means your users will likely connect to a call through us-west-2 instead. This failover happens automatically and transparently, and you don't need to do anything. We've already scaled up call servers in us-west-2 to handle the increase in sessions. We'll keep you posted as we continue to monitor the situation.Oct <var data-var='date'>20</var>, <var data-var='time'>12:57</var> UTC Identified - AWS is addressing an issue with DynamoDB that is causing problems with many of their services in us-east-1. We have seen issues loading the dashboard and docs sites, and for a few hours we've been unable to log in to Statuspage to update this site. We're sorry about that delay. Up until a few minutes ago, our metrics suggested that core call functionality wasn't being impacted. We just saw a small number of call failures in us-east-1, though. We're continuing to monitor the situation and do what we can to minimize this outage's impact on you and your customers. We'll post more information here as we have it.

Impact from the ongoing AWS us-east-1 outage

Sep <var data-var='date'>25</var>, <var data-var='time'>21:22</var> UTC Resolved - This incident has been resolved.Sep <var data-var='date'>24</var>, <var data-var='time'>18:17</var> UTC Investigating - We've temporarily disabled displaying the API and webhook log views on the Dashboard Developer page. The Daily API is still fully operational, and these logs are still being collected. We expect to re-enable these log views in the next few hours. If you have an urgent need for this data, please contact help@daily.co.

Dashboard API and webhook logs temporarily disabled

Sep <var data-var='date'> 4</var>, <var data-var='time'>16:45</var> UTC Resolved - This incident has been resolved.Sep <var data-var='date'> 4</var>, <var data-var='time'>14:48</var> UTC Investigating - We're experiencing delays in making log data available via the /logs REST API endpoint. We're not losing any log data, but the process that ingests logs for the REST API is bottlenecked. This means that the most recent data available from the /logs endpoint is about 6 hours old. We'll update this incident once the situation has been resolved.

Delayed data from the /logs API endpoint

Jul <var data-var='date'>17</var>, <var data-var='time'>21:40</var> UTC Resolved - This incident has been resolved.Jul <var data-var='date'>17</var>, <var data-var='time'>21:07</var> UTC Monitoring - We've rolled back a server release, and the SIP error messages have stopped. We're monitoring for any further problems.Jul <var data-var='date'>17</var>, <var data-var='time'>21:01</var> UTC Identified - We're in the process of rolling back a release to address this issue.Jul <var data-var='date'>17</var>, <var data-var='time'>20:52</var> UTC Investigating - We're investigating reports of SIP/PSTN dialin and dialout failing, or stopping immediately after starting.

Issues with SIP dialin/dialout

Jul <var data-var='date'>14</var>, <var data-var='time'>17:25</var> UTC Resolved - We're continuing to investigate the root cause, but we've resolved all impact to our running services.Jul <var data-var='date'>14</var>, <var data-var='time'>17:06</var> UTC Identified - We're aware of an infrastructure issue that may be causing some intermittent failures for Daily API requests.Jul <var data-var='date'>14</var>, <var data-var='time'>16:43</var> UTC Investigating - We're seeing alerts that webhooks are taking longer than normal to deliver.

Jun <var data-var='date'>12</var>, <var data-var='time'>21:47</var> UTC Resolved - This incident has been resolved.Jun <var data-var='date'>12</var>, <var data-var='time'>20:49</var> UTC Update - We had a brief spike in response time, but platform operations appear normal since approximately 20:05 UTC. We're continuing to monitor for any follow-on effects, and we'll leave this status post open until various upstream providers close their respective status incidents.Jun <var data-var='date'>12</var>, <var data-var='time'>20:06</var> UTC Update - We're seeing some increases in response time from our internal API endpoint that returns connection information to daily-js. This may be causing "room lookup timed out" errors for users attempting to join calls. We believe this is related to the ongoing GCP incident.Jun <var data-var='date'>12</var>, <var data-var='time'>18:56</var> UTC Identified - This appears to be related to the Google Cloud outage posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW We'll post here as we learn more.Jun <var data-var='date'>12</var>, <var data-var='time'>18:39</var> UTC Investigating - Daily is being affected by widespread internet outages in authentication services. This is currently affecting our ability to collect or view some call metrics. We'll post more as soon as we have more information.

Issues with logging and telemetry

Apr <var data-var='date'>17</var>, <var data-var='time'>20:36</var> UTC Resolved - This incident has been resolved.Apr <var data-var='date'>17</var>, <var data-var='time'>20:11</var> UTC Monitoring - It appears that SignalWire rolled back the change, and the errors have stopped. We are monitoring for any further issues.Apr <var data-var='date'>17</var>, <var data-var='time'>19:57</var> UTC Identified - SignalWire has confirmed that they made an unannounced change that's causing the problem, and they're working to revert it. We're also working on a quick production update to work around the issue if they can't revert quickly enough.Apr <var data-var='date'>17</var>, <var data-var='time'>19:33</var> UTC Investigating - We're investigating elevated error rates from SignalWire when provisioning SIP/PSTN resources. If you're creating rooms with dial-in or dial-out enabled and it isn't absolutely necessary, you can remove those params from your room creation request to successfully create rooms.

Elevated error rates for SIP/PSTN

Apr <var data-var='date'>15</var>, <var data-var='time'>21:56</var> UTC Resolved - This incident has been resolved.Apr <var data-var='date'>15</var>, <var data-var='time'>21:33</var> UTC Monitoring - We've deployed an update to production servers to remediate the issue, and we're testing to ensure everything is fixed.Apr <var data-var='date'>15</var>, <var data-var='time'>19:38</var> UTC Update - We've identified an issue with how Daily and SignalWire are negotiating audio codecs for incoming SIP/PSTN calls. We're working on an update that will change the default audio codec used between Daily and SignalWire to Opus. Once this is deployed, you may still experience audio issues if you've explicitly set your SIP audio codec to PCMU as described here: https://docs.daily.co/guides/products/dial-in-dial-out/sip#sip-dial-in-audio-and-videoApr <var data-var='date'>15</var>, <var data-var='time'>16:06</var> UTC Identified - We've identified an issue causing degraded ("broken up" or "choppy") audio between some Daily sessions and SignalWire SIP/PSTN endpoints. PSTN dialout and PIN dialin do not seem to be affected. SIP dialout seems to be moderately affected. PIN-less PSTN dialin and SIP dialin seem to be experiencing a much higher proportion of affected calls.Apr <var data-var='date'>15</var>, <var data-var='time'>14:11</var> UTC Investigating - We're investigating reports of poor quality audio for SIP/PSTN participants in some calls.

Issues with SIP/PSTN audio quality

Apr <var data-var='date'> 9</var>, <var data-var='time'>21:48</var> UTC Resolved - This incident has been resolved.Apr <var data-var='date'> 9</var>, <var data-var='time'>20:12</var> UTC Update - We're still waiting on some additional fixes from Signalwire. Room creation with dial-in/dial-out is working, but you still may experience problems updating dial-in/dial-out settings for existing rooms. Daily Bots and Pipecat Cloud users should be unaffected.Apr <var data-var='date'> 9</var>, <var data-var='time'>18:53</var> UTC Monitoring - Signalwire has restored service, but we're still seeing some error responses from their API. We believe the majority of room creation failure issues have been resolved. We'll post here again when we've handled this last issue.Apr <var data-var='date'> 9</var>, <var data-var='time'>17:46</var> UTC Update - We've been told we should be back online after a hotfix at approximately 18:00 UTC, or 15 minutes from now. We'll post another update as soon as we have more info.Apr <var data-var='date'> 9</var>, <var data-var='time'>17:22</var> UTC Update - We're still waiting on Signalwire to restore service.Apr <var data-var='date'> 9</var>, <var data-var='time'>14:26</var> UTC Update - We're still monitoring. In addition to dialout_enabled, you'll need to remove other room properties related to SIP/PSTN, dialin, and/or dialout to create rooms.Apr <var data-var='date'> 9</var>, <var data-var='time'>13:50</var> UTC Update - We're continuing to monitor an issue from our SIP/PSTN provider. The rest of the Daily platform is unaffected. If you're getting an error when trying to create a room, you can remove the dialout_enabled property from the room creation request and try again.Apr <var data-var='date'> 9</var>, <var data-var='time'>12:53</var> UTC Identified - Our SIP/PSTN provider has identified an issue and they're deploying a fix. In the meantime, you should still be able to create Daily rooms without provisioning SIP/PSTN.Apr <var data-var='date'> 9</var>, <var data-var='time'>12:15</var> UTC Investigating - We're seeing elevated error rates from our SIP/PSTN provider. If you're creating rooms with dial-in support and getting errors, you may want to retry creating those rooms without dial-in, and then add dial-in with an update REST request.

Elevated SIP/PSTN error rates

Mar <var data-var='date'>12</var>, <var data-var='time'>20:46</var> UTC Resolved - This incident has been resolved.Mar <var data-var='date'>12</var>, <var data-var='time'>20:26</var> UTC Monitoring - We've deployed a fix, and we're monitoring for any further issues.Mar <var data-var='date'>12</var>, <var data-var='time'>18:45</var> UTC Update - We've identified the issue, and we're testing a fix in our staging infrastructure. We'll post another update when we've deployed the fix.Mar <var data-var='date'>12</var>, <var data-var='time'>15:59</var> UTC Identified - We're getting reports of delayed audio from some customers using SIP dialin and dialout. When the phone user joins the call, they can talk and others will hear them, but the phone user won't hear any audio from other call participants (bot or human) for the first 20-30 seconds. We're continuing to troubleshoot the issue, and we'll post here as soon as we have more info.Mar <var data-var='date'>12</var>, <var data-var='time'>15:19</var> UTC Investigating - We're investigating an issue that's causing delays in audio connection for a some SIP/PSTN calls.

Delayed audio for some SIP/PSTN dial-in calls

Jan <var data-var='date'>28</var>, <var data-var='time'>16:58</var> UTC Resolved - This issue has been resolved.Jan <var data-var='date'>28</var>, <var data-var='time'>16:36</var> UTC Monitoring - We've resolved the issue and we're monitoring to ensure the platform is operating normally.Jan <var data-var='date'>28</var>, <var data-var='time'>16:21</var> UTC Investigating - We're investigating an issue that may be preventing some users from joining meeting rooms.

Issues connecting to rooms

Nov <var data-var='date'>25</var>, <var data-var='time'>19:26</var> UTC Resolved - We've deployed an update that increases the throughput of the database that was the bottleneck in today's incident. We'll have more info about additional remediations and a postmortem for today's incident within the next few days.Nov <var data-var='date'>25</var>, <var data-var='time'>18:08</var> UTC Update - Our metrics have stayed at normal levels since our remediating actions about 30 minutes ago. We're continuing to monitor the platform while we discuss longer-term solutions to make absolutely sure we've addressed the root cause here.Nov <var data-var='date'>25</var>, <var data-var='time'>17:33</var> UTC Monitoring - We've made some changes to the affected database, and our metrics and error rates have returned to normal. We've also re-enabled delivery of all webhooks, and we're monitoring for any further issues.Nov <var data-var='date'>25</var>, <var data-var='time'>17:25</var> UTC Update - We're addressing an issue with an internal database that's causing problems with existing meetings, as well as starting new ones. Your users are likely seeing some failures when trying to join meeting sessions, and users in ongoing sessions are seeing occasional meeting moves. We'll post more information as soon as soon as it's available.Nov <var data-var='date'>25</var>, <var data-var='time'>17:10</var> UTC Identified - We've temporarily disabled the component that sends webhooks.Nov <var data-var='date'>25</var>, <var data-var='time'>17:07</var> UTC Update - We're continuing to investigate the source of meeting disruptions. Customers may be experiencing 'meeting moves' where a call session moves from one server to another, causing a 2-3 second disruption to the call. You may also see delays in receiving meeting.started and meeting.ended webhooks.Nov <var data-var='date'>25</var>, <var data-var='time'>16:48</var> UTC Investigating - We're investigating issues that may be causing problems with network connections between regions.

Networking issues

Oct <var data-var='date'>30</var>, <var data-var='time'>21:03</var> UTC Resolved - We're still making a few small infrastructure changes, but our internal metrics have been back at normal levels for some time.Oct <var data-var='date'>30</var>, <var data-var='time'>18:59</var> UTC Identified - We've identified an issue causing some recordings to fail to start, specifically in the Oracle Cloud San Jose region. We've already made some infrastructure changes that should be routing new recording requests to other regions. If you've seen a recording fail to start, you can try starting it again using daily-js or the REST API. We'll keep you posted on our progress resolving the issue.

Issues starting recordings

Oct <var data-var='date'>22</var>, <var data-var='time'>22:02</var> UTC Resolved - This issue has been resolved. We will post more information about this incident in the near future.Oct <var data-var='date'>22</var>, <var data-var='time'>20:59</var> UTC Monitoring - API latency and errors have stayed at normal levels for a while now, but we're continuing to monitor for any further impact.Oct <var data-var='date'>22</var>, <var data-var='time'>19:59</var> UTC Update - API error levels have decreased considerably, but we're still working on full remediation. More updates to come.Oct <var data-var='date'>22</var>, <var data-var='time'>19:14</var> UTC Identified - We've identified an issue causing some slowdowns in one of our databases, leading to some delayed or failed API responses. We've solved the root cause of the issue, but we're being cautious about restoring the database to full functionality, so we expect the delays to continue for a short time.Oct <var data-var='date'>22</var>, <var data-var='time'>18:46</var> UTC Investigating - We're investigating an issue that's causing delays with some API operations, such as creating rooms and starting recordings. We'll post more info as soon as we have it.

Delayed API calls

Oct <var data-var='date'>22</var>, <var data-var='time'>02:38</var> UTC Resolved - This incident has been resolved. Customers needing assistance with missing webhook deliveries should contact support via help@daily.co.Oct <var data-var='date'>21</var>, <var data-var='time'>17:50</var> UTC Monitoring - Between 14:01 UTC and 17:12 UTC webhooks for meeting.started and meeting.ended events were not delivered. We have applied a mitigation and are continuing to monitor. The underlying cause for the missing deliveries is still under investigation.

Missing meeting webhook deliveries

Aug <var data-var='date'> 7</var>, <var data-var='time'>20:17</var> UTC Resolved - This incident has been resolved.Aug <var data-var='date'> 7</var>, <var data-var='time'>19:49</var> UTC Update - The upstream issue has been resolved, and we're monitoring for any more issues.Aug <var data-var='date'> 7</var>, <var data-var='time'>19:49</var> UTC Monitoring - The upstream issue has been resolved, and we're monitoring for any more issues.Aug <var data-var='date'> 7</var>, <var data-var='time'>19:41</var> UTC Investigating - Some customers are seeing 400 BAD_REQUEST messages when trying to load dashboard.daily.co. This is likely related to a Vercel incident: https://www.vercel-status.com/incidents/f6b2blrl5f5f

Is Daily down?

Daily status is UP

Lastest published events

Issues with inbound phone calls

Webhook delivery delays

SIP / PSTN Disruption

Daily Services, Components and Regions

Recent Daily events

Issues with inbound phone calls

Webhook delivery delays

SIP / PSTN Disruption

Increase in "Failed to load call object bundle" errors

Issues with transcriptions and captions

Webhook deliveries delayed

Get your FREE account now,
5 monitors included!

Is Daily down?

Daily status is UP

Lastest published events

Issues with inbound phone calls

Webhook delivery delays

SIP / PSTN Disruption

Daily Services, Components and Regions

Recent Daily events

Issues with inbound phone calls

Webhook delivery delays

SIP / PSTN Disruption

Increase in "Failed to load call object bundle" errors

Issues with transcriptions and captions

Webhook deliveries delayed

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!