Aug <var data-var='date'>18</var>, <var data-var='time'>16:31</var> UTC Resolved - This incident has been resolved, export jobs are complete Aug <var data-var='date'>17</var>, <var data-var='time'>20:17</var> UTC Update - Export jobs are being run for the next 24 hours, beginning with 8/13. We will notify here when the jobs are complete and exports are available. Aug <var data-var='date'>17</var>, <var data-var='time'>15:25</var> UTC Identified - The issue has been identified and we have started the process of running delayed exports jobs. Aug <var data-var='date'>17</var>, <var data-var='time'>14:55</var> UTC Investigating - We are investigating an issue where CSV exports are delayed beginning 8/13 to today.

Mux Data CSV Exports Delayed

Jul <var data-var='date'>21</var>, <var data-var='time'>01:15</var> UTC Resolved - The fix resolved the issue and asset creation back to operational. Jul <var data-var='date'>21</var>, <var data-var='time'>01:08</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>21</var>, <var data-var='time'>00:57</var> UTC Investigating - We are currently investigating slowness in asset creation that affects limited number of assets in one of our North America cloud locations.

Asset Creation Slowness in Certain Region

Jul <var data-var='date'>16</var>, <var data-var='time'>17:07</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>16</var>, <var data-var='time'>00:21</var> UTC Update - All customer impact has been mitigated. Traffic remains directed away from the affected infrastructure while we work with our cloud provider on the underlying issue. We will remain in this state until we have fully validated the issue is resolved and we can safely restore normal operation. Once normal operation is restored, we will update and close this incident accordingly. Jul <var data-var='date'>15</var>, <var data-var='time'>21:06</var> UTC Update - We have identified service disruption in underlying cloud infrastructure in a single availability zone, which impacted a portion of video ingestion and playback. We have routed traffic away from the affected infrastructure and are continuing to monitor while we work with the service provider on resolution. Jul <var data-var='date'>15</var>, <var data-var='time'>21:06</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>15</var>, <var data-var='time'>19:13</var> UTC Identified - Customers may experience issues with video live and/or playback. Root cause has been identified and fix is being worked on. More updates to come as we have it.

video live and playback issues

Jul <var data-var='date'>14</var>, <var data-var='time'>20:01</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>14</var>, <var data-var='time'>18:03</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>14</var>, <var data-var='time'>17:43</var> UTC Update - We are continuing to work on a fix for this issue. Jul <var data-var='date'>14</var>, <var data-var='time'>17:13</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'>14</var>, <var data-var='time'>17:09</var> UTC Investigating - We are investigating reports of video playback failures. Some customers may experience playback errors or timeouts.

Video Playback Issues

Jul <var data-var='date'>14</var>, <var data-var='time'>08:46</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>14</var>, <var data-var='time'>00:43</var> UTC Monitoring - A fix has been implemented and we are monitoring the situation Jul <var data-var='date'>13</var>, <var data-var='time'>21:33</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'>13</var>, <var data-var='time'>21:11</var> UTC Update - We are continuing to investigate this issue. Jul <var data-var='date'>13</var>, <var data-var='time'>20:42</var> UTC Investigating - We are investigating an issue affecting live streams. Some viewers may experience dropped or interrupted live streams. We will provide updates as more information becomes available.

Live stream disruptions

Jul <var data-var='date'> 9</var>, <var data-var='time'>14:02</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 8</var>, <var data-var='time'>19:57</var> UTC Update - We are continuing to monitor for any further issues. Jul <var data-var='date'> 8</var>, <var data-var='time'>19:57</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'> 8</var>, <var data-var='time'>17:43</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'> 8</var>, <var data-var='time'>17:29</var> UTC Investigating - We are investigating an issue affecting live streams. Some viewers may experience dropped or interrupted live streams. Our team has identified the underlying infrastructure issue and is actively working on a resolution. We will provide updates as more information becomes available.

Live Stream Disruptions

Jul <var data-var='date'> 2</var>, <var data-var='time'>01:13</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 1</var>, <var data-var='time'>22:50</var> UTC Monitoring - A fix has been implemented and we are monitoring results. Delayed data was for a subset of customers between 22:12 UTC - 22:44 UTC. No data loss. Jul <var data-var='date'> 1</var>, <var data-var='time'>22:35</var> UTC Investigating - We are investigating an issue with collecting Mux data beacons. Beginning 22:12 UTC, view session data will be delayed.

Mux Data beacon collection issues

Jun <var data-var='date'>26</var>, <var data-var='time'>18:48</var> UTC Resolved - No further issues observed during monitoring period. Live streams are operational. Jun <var data-var='date'>26</var>, <var data-var='time'>16:50</var> UTC Monitoring - Between 16:15 and 16:20 UTC a small number of live streams were disconnected. Active live streams were able to reconnect and resume, and there is no long term impact to DVR or VOD content from affected live streams other than resuming on a new Asset ID. All systems are currently operational and we are monitoring the live ingest systems.

Live Stream Disconnects

Jun <var data-var='date'>25</var>, <var data-var='time'>16:48</var> UTC Resolved - A small number of live streams were disconnected around 13:10 UTC, ending the corresponding Asset. Active live streams were able to reconnect and resume, and there is no long term impact to DVR or VOD content from affected live streams other than resuming on a new Asset ID. All systems are currently green. Jun <var data-var='date'>25</var>, <var data-var='time'>13:56</var> UTC Investigating - We are investigating an incident which caused the disconnection of multiple live streams at 13:11 UTC.

Live stream disconnections

Jun <var data-var='date'>18</var>, <var data-var='time'>13:18</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>18</var>, <var data-var='time'>11:31</var> UTC Monitoring - Video delivery for Live and VOD in the impacted region has returned to normal operations.  The impacted time frame was 9:30 - 10:30 UTC.  We are continuing to monitor. Jun <var data-var='date'>18</var>, <var data-var='time'>10:57</var> UTC Identified - We've identified an issue with a CDN provider in the impacted regions. We are seeing recovery, and continuing to monitor. Jun <var data-var='date'>18</var>, <var data-var='time'>10:16</var> UTC Investigating - We are experiencing increased errors with delivery across Live and VOD in Europe regions  and are actively investigating.

Increased error in Video Live and VOD

Jun <var data-var='date'>10</var>, <var data-var='time'>18:58</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>10</var>, <var data-var='time'>16:11</var> UTC Monitoring - A fix has been implemented and we are monitoring. Jun <var data-var='date'>10</var>, <var data-var='time'>15:10</var> UTC Identified - We have identified the issue and are implementing a fix. Jun <var data-var='date'>10</var>, <var data-var='time'>14:04</var> UTC Investigating - We are investigating an issue causing an increase in errors in the Mux Data dashboard and API.

Elevated errors in Mux Data dashboard and API

May <var data-var='date'>19</var>, <var data-var='time'>23:56</var> UTC Resolved - From 21:02 UTC to 22:55 UTC some live streams may have experienced elevated rebuffering. We have identified and mitigated the issue.

Elevated rebuffering in live streams

May <var data-var='date'>19</var>, <var data-var='time'>21:00</var> UTC Completed - The scheduled maintenance has been completed. May <var data-var='date'>19</var>, <var data-var='time'>20:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>12</var>, <var data-var='time'>22:10</var> UTC Scheduled - In order to improve the reliability of our realtime systems, we will be undergoing scheduled maintenance. During this time, customers from EU may see duplicate messages via our view stream exports to their Kinesis or PubSub endpoints. This will not affect views or metrics once this window is complete.

Mux Data Infrastructure Maintenance

May <var data-var='date'>14</var>, <var data-var='time'>20:57</var> UTC Resolved - This incident has been resolved. May <var data-var='date'>14</var>, <var data-var='time'>20:33</var> UTC Monitoring - We have identified the issue and have deployed a fix. We are continuing to monitor. May <var data-var='date'>14</var>, <var data-var='time'>20:31</var> UTC Investigating - We are seeing high webhook delivery latency. We are actively investigating

Webhook Delivery Stalled

May <var data-var='date'> 8</var>, <var data-var='time'>03:44</var> UTC Resolved - From 03:08UTC to 03:14UTC we saw increased request error rates and webhook delivery latency increases. The issue is mitigated and systems are operating normally.

Elevated API errors and webhook delivery latency

May <var data-var='date'> 7</var>, <var data-var='time'>23:41</var> UTC Resolved - This incident has been resolved May <var data-var='date'> 7</var>, <var data-var='time'>23:18</var> UTC Monitoring - Our team has implemented a fix and we are monitoring the results. We’ll post an update as soon as the issue is fully resolved. May <var data-var='date'> 7</var>, <var data-var='time'>22:39</var> UTC Identified - Our team has identified the cause of elevated Data Historical metrics errors and is actively working on a fix. We’ll continue to share updates as we have them. May <var data-var='date'> 7</var>, <var data-var='time'>22:32</var> UTC Investigating - We are currently investigating errors in Mux Data historical metrics beginning at approximately 22:08 UTC. We apologize for the inconvenience and will share an update once we have more information.

Elevated errors in Mux Data Historical metrics

May <var data-var='date'> 6</var>, <var data-var='time'>18:24</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 6</var>, <var data-var='time'>17:42</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'> 6</var>, <var data-var='time'>16:41</var> UTC Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'> 6</var>, <var data-var='time'>16:36</var> UTC Investigating - We are currently investigating an issue where playback may experience elevated rebuffering rates and playback failures.

Elevated rebuffering and playback failures

May <var data-var='date'> 5</var>, <var data-var='time'>20:54</var> UTC Resolved - This incident has been resolved. May <var data-var='date'> 5</var>, <var data-var='time'>19:50</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'> 5</var>, <var data-var='time'>19:18</var> UTC Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'> 5</var>, <var data-var='time'>18:52</var> UTC Investigating - We are currently investigating the issue

Elevated rebuffering and startup times for some on-demand assets and live streams

Apr <var data-var='date'>30</var>, <var data-var='time'>03:30</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'>30</var>, <var data-var='time'>02:26</var> UTC Monitoring - Our pipeline latency is resolved and we are caught up as of 1:50 UTC Apr <var data-var='date'>30</var>, <var data-var='time'>01:03</var> UTC Identified - The issue has been identified and a fix is being implemented. Apr <var data-var='date'>30</var>, <var data-var='time'>00:22</var> UTC Investigating - Mux Data Views for EU customers are delayed beginning 22:53 UTC. We are investigating

Mux Data Delayed Views for EU customers

Apr <var data-var='date'>28</var>, <var data-var='time'>17:33</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'>28</var>, <var data-var='time'>16:22</var> UTC Monitoring - We are monitoring for additional latency. EU saw elevated latency 15:50 UTC to 16:04 UTC. Monitoring data was inaccurate during that window and views were delayed. No data was lost. Apr <var data-var='date'>28</var>, <var data-var='time'>16:05</var> UTC Investigating - We are seeing some latency in both Monitoring and Historical views starting 15:53 UTC for EU customers and are currently investigating

Mux Data Pipeline Latency for EU Customers

Apr <var data-var='date'>28</var>, <var data-var='time'>03:50</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'>27</var>, <var data-var='time'>20:09</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>27</var>, <var data-var='time'>20:08</var> UTC Investigating - Small percentage of static rendition downloads. We are currently investigating this issue.

Issues downloading MP4 assets

Apr <var data-var='date'>22</var>, <var data-var='time'>19:46</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'>22</var>, <var data-var='time'>18:52</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>22</var>, <var data-var='time'>18:10</var> UTC Investigating - We are investigating increased startup times, re-buffering, and playback errors from 429 errors on some live streams and on-demand assets.

Increased re-buffering and errors for some assets and live streams

Apr <var data-var='date'>22</var>, <var data-var='time'>00:06</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'>21</var>, <var data-var='time'>18:20</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>21</var>, <var data-var='time'>18:14</var> UTC Identified - The issue has been identified and a fix is being implemented. Apr <var data-var='date'>21</var>, <var data-var='time'>17:52</var> UTC Update - We are investigating an issue causing delays in processing for some assets. Additionally, the generation of static renditions, masters, and captions may be delayed. Apr <var data-var='date'>21</var>, <var data-var='time'>17:36</var> UTC Investigating - We are investigating an issue causing delays in processing for some assets.

Ingest delays for some assets

Apr <var data-var='date'> 9</var>, <var data-var='time'>15:49</var> UTC Resolved - This incident has been resolved. Apr <var data-var='date'> 9</var>, <var data-var='time'>14:27</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'> 9</var>, <var data-var='time'>14:20</var> UTC Investigating - We are currently investigating an issue where dashboard.mux.com is unavailable and may be showing an exception when loaded. All Data and Video services aside from dashboard.mux.com are unaffected.

Mux Dashboard unavailable

Mar <var data-var='date'> 7</var>, <var data-var='time'>01:57</var> UTC Resolved - Engineers have resolved the issue with clips not having audio. Any new clips will function as expected. Impacted clips created during the incident window are not retroactively corrected, but can be re-created from the original asset. Mar <var data-var='date'> 6</var>, <var data-var='time'>22:57</var> UTC Identified - A recent encoding change caused a mismatch in initialization metadata for some clips that were created in the last 24 hours. This would affect clips of assets created before Thursday, March 6 — 15:11 UTC that utilized "basic" or "premium" video_quality. A fix is currently being worked on.

Is mux down?

mux status is UP

Lastest published events

Mux Data CSV Exports Delayed

Asset Creation Slowness in Certain Region

video live and playback issues

mux Services, Components and Regions

Recent mux events

Mux Data CSV Exports Delayed

Asset Creation Slowness in Certain Region

video live and playback issues

Video Playback Issues

Live stream disruptions

Live Stream Disruptions