Mar <var data-var='date'>16</var>, <var data-var='time'>16:15</var> MDT Resolved - This incident has been resolved. Mar <var data-var='date'>16</var>, <var data-var='time'>16:02</var> MDT Investigating - We are experiencing delays in our ingest pipeline

Ingest Delays

Aug <var data-var='date'>24</var>, <var data-var='time'>09:23</var> MDT Resolved - This incident has been resolved. Aug <var data-var='date'>24</var>, <var data-var='time'>08:44</var> MDT Investigating - A subset of customers are experiencing ingest delays. Investigating.

Aug <var data-var='date'> 6</var>, <var data-var='time'>04:09</var> MDT Resolved - This incident has been resolved. Aug <var data-var='date'> 6</var>, <var data-var='time'>04:06</var> MDT Monitoring - Metrics are recovering. Aug <var data-var='date'> 6</var>, <var data-var='time'>03:22</var> MDT Identified - Certain customers are unable to view the UI and metrics are delayed.

Ingest delays, emergency maintenance

Aug <var data-var='date'> 5</var>, <var data-var='time'>14:14</var> MDT Resolved - This incident has been resolved. Aug <var data-var='date'> 5</var>, <var data-var='time'>10:52</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'> 5</var>, <var data-var='time'>10:34</var> MDT Identified - We are seeing problematic database behavior and working to mitigate it.

Ingest delays, UI latency

Jul <var data-var='date'>31</var>, <var data-var='time'>16:49</var> MDT Resolved - This incident has been resolved. Jul <var data-var='date'>31</var>, <var data-var='time'>15:05</var> MDT Monitoring - We have made some changes to mitigate the situation. Metrics for affected customers are recovering. Jul <var data-var='date'>31</var>, <var data-var='time'>13:55</var> MDT Update - A portion of customers are seeing UI unavailability and slow metric ingest times. We are still working towards a sustainable solution. Jul <var data-var='date'>31</var>, <var data-var='time'>12:05</var> MDT Identified - We are working to address high load on database instances. We are still receiving data as normal.

Ingest delays and UI latency

Jul <var data-var='date'> 1</var>, <var data-var='time'>12:30</var> MDT Resolved - This incident has been resolved. Jul <var data-var='date'> 1</var>, <var data-var='time'>11:39</var> MDT Monitoring - Ingested metrics are catching up. Jul <var data-var='date'> 1</var>, <var data-var='time'>11:22</var> MDT Identified - We are having some database issues affecting a portion of accounts. Applying a fix. Metrics are still being ingested.

Database performance issues

Jun <var data-var='date'>25</var>, <var data-var='time'>21:28</var> MDT Resolved - Some metrics are still delayed, but backfill is proceeding. Jun <var data-var='date'>25</var>, <var data-var='time'>21:07</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>25</var>, <var data-var='time'>20:28</var> MDT Investigating - We are performing emergency maintenance on our database. Some customers are unfortunately being impacted.

Database maintenance

Jun <var data-var='date'>25</var>, <var data-var='time'>15:28</var> MDT Resolved - This incident has been resolved. Jun <var data-var='date'>25</var>, <var data-var='time'>14:45</var> MDT Monitoring - Remaining customers are now recovering. Jun <var data-var='date'>25</var>, <var data-var='time'>14:36</var> MDT Identified - Most customers are recovered but a portion are still impacted. Jun <var data-var='date'>25</var>, <var data-var='time'>14:18</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>25</var>, <var data-var='time'>13:49</var> MDT Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'>25</var>, <var data-var='time'>13:29</var> MDT Investigating - We are currently investigating this issue.

Ingest delays

Jun <var data-var='date'>17</var>, <var data-var='time'>08:53</var> MDT Resolved - This incident has been resolved. Jun <var data-var='date'>17</var>, <var data-var='time'>07:11</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>17</var>, <var data-var='time'>06:47</var> MDT Identified - We are suffering some cascading effects that are impacting web portal availability as well. Jun <var data-var='date'>17</var>, <var data-var='time'>05:56</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>17</var>, <var data-var='time'>04:46</var> MDT Investigating - We are seeing delays in processing metric information. We are investigating.

Jun <var data-var='date'> 3</var>, <var data-var='time'>20:30</var> MDT Resolved - A subset of customers experienced a delay in metrics. All metrics have caught up at this point.

Partial Outage - Delay in Metrics

May <var data-var='date'>25</var>, <var data-var='time'>16:13</var> MDT Resolved - This incident has been resolved. May <var data-var='date'>25</var>, <var data-var='time'>14:27</var> MDT Investigating - We are seeing a spike in metrics, causing delays

May <var data-var='date'>22</var>, <var data-var='time'>20:27</var> MDT Resolved - This incident has been resolved. May <var data-var='date'>22</var>, <var data-var='time'>15:05</var> MDT Monitoring - Zookeeper corruption has been rooted out. Things appear healthier and catching up in all cases. May <var data-var='date'>22</var>, <var data-var='time'>11:43</var> MDT Update - We are not to full resolution yet. May <var data-var='date'>21</var>, <var data-var='time'>23:42</var> MDT Update - Throughput has improved although behavior of individual partitions remains a problem and is still causing delays in some cases. May <var data-var='date'>21</var>, <var data-var='time'>20:13</var> MDT Identified - It has been a long day with kafka. We continue to experience instability, causing lag and dropped payloads. May <var data-var='date'>21</var>, <var data-var='time'>10:13</var> MDT Monitoring - An alternate approach has been applied, we are watching. May <var data-var='date'>21</var>, <var data-var='time'>09:08</var> MDT Identified - The initial fix was unsuccessful, certain accounts are now substantially delayed. May <var data-var='date'>21</var>, <var data-var='time'>08:33</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>21</var>, <var data-var='time'>07:55</var> MDT Investigating - We are currently investigating this issue.

Apr <var data-var='date'>24</var>, <var data-var='time'>11:15</var> MDT Resolved - This incident has been resolved. Apr <var data-var='date'>24</var>, <var data-var='time'>11:01</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>24</var>, <var data-var='time'>09:43</var> MDT Identified - Metric ingestion is being delayed for a set of users. We are working towards resolution.

Apr <var data-var='date'>15</var>, <var data-var='time'>20:06</var> MDT Resolved - This incident has been resolved. Apr <var data-var='date'>15</var>, <var data-var='time'>08:10</var> MDT Update - Ingest is recovering. Some accounts will require additional backfill of data, which we are working on. Apr <var data-var='date'>15</var>, <var data-var='time'>07:18</var> MDT Identified - We have identified the issue and are working on fixes. Apr <var data-var='date'>15</var>, <var data-var='time'>06:38</var> MDT Update - We are continuing to investigate this issue. Apr <var data-var='date'>15</var>, <var data-var='time'>06:37</var> MDT Investigating - Ingested records are taking longer than usual to process. In some cases, this is affecting alerting.

Ingest delays and performance degradation

Apr <var data-var='date'> 1</var>, <var data-var='time'>11:03</var> MDT Resolved - This incident has been resolved. Apr <var data-var='date'> 1</var>, <var data-var='time'>10:16</var> MDT Monitoring - A fix has been implemented and we are monitoring the results.

Mar <var data-var='date'>13</var>, <var data-var='time'>11:23</var> MDT Resolved - This incident has been resolved. Mar <var data-var='date'>13</var>, <var data-var='time'>10:13</var> MDT Monitoring - Delays are recovering, we are monitoring. Mar <var data-var='date'>13</var>, <var data-var='time'>09:56</var> MDT Identified - The issue has been identified and a fix is being implemented. Mar <var data-var='date'>13</var>, <var data-var='time'>09:47</var> MDT Investigating - Ingested data is suffering delays in processing. Our team is working on a fix.

Aug <var data-var='date'>19</var>, <var data-var='time'>12:09</var> MDT Resolved - This incident has been resolved. Aug <var data-var='date'>19</var>, <var data-var='time'>12:09</var> MDT Update - We are continuing to monitor for any further issues. Aug <var data-var='date'>19</var>, <var data-var='time'>09:31</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>19</var>, <var data-var='time'>08:50</var> MDT Update - We are continuing to investigate this issue. Aug <var data-var='date'>19</var>, <var data-var='time'>08:50</var> MDT Investigating - We are currently investigating this issue.

Ingest delays and dashboard degredation

Feb <var data-var='date'>22</var>, <var data-var='time'>15:37</var> MST Resolved - This incident has been resolved. Feb <var data-var='date'>22</var>, <var data-var='time'>12:02</var> MST Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>22</var>, <var data-var='time'>11:44</var> MST Identified - The issue has been identified and a fix is being implemented.

Data Ingest Issues

Feb <var data-var='date'>15</var>, <var data-var='time'>18:10</var> MST Resolved - This incident has been resolved. Feb <var data-var='date'>15</var>, <var data-var='time'>17:54</var> MST Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'> 8</var>, <var data-var='time'>21:12</var> MST Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 8</var>, <var data-var='time'>20:59</var> MST Update - Our apologies but maintenance has run long. We anticipate completing shortly. Feb <var data-var='date'> 8</var>, <var data-var='time'>20:00</var> MST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>30</var>, <var data-var='time'>10:00</var> MST Scheduled - We will be performing database updates during this window. There will be a brief period where the Scout UI is unavailable. We will still receive data from Scout Agents, but processing will be delayed.

Database Updates

Aug <var data-var='date'> 8</var>, <var data-var='time'>19:34</var> MDT Resolved - This incident has been resolved. Aug <var data-var='date'> 8</var>, <var data-var='time'>18:14</var> MDT Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'> 8</var>, <var data-var='time'>18:08</var> MDT Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'> 8</var>, <var data-var='time'>16:48</var> MDT Investigating - We are currently investigating this issue.

Jun <var data-var='date'> 9</var>, <var data-var='time'>20:17</var> MDT Resolved - This incident has been resolved. Jun <var data-var='date'> 9</var>, <var data-var='time'>17:38</var> MDT Monitoring - We have implemented a fix and are monitoring the issue. Jun <var data-var='date'> 9</var>, <var data-var='time'>16:24</var> MDT Investigating - We are currently investigating this issue.

Ingest Issues

May <var data-var='date'>11</var>, <var data-var='time'>16:31</var> MDT Completed - The scheduled maintenance has been completed. May <var data-var='date'>11</var>, <var data-var='time'>16:00</var> MDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>11</var>, <var data-var='time'>14:34</var> MDT Scheduled - We will be performing maintenance during this time. There may be brief periods of UI interruption and lag in ingested metrics.

Database Maintenance

Apr <var data-var='date'>21</var>, <var data-var='time'>12:47</var> MDT Resolved - This incident has been resolved. Apr <var data-var='date'>21</var>, <var data-var='time'>12:20</var> MDT Monitoring - Ingested metrics are being processed, lag is recovering. Apr <var data-var='date'>21</var>, <var data-var='time'>12:07</var> MDT Investigating - Data ingest is lagging. Some users are experiencing UI errors.

Ingest Delays and UI Degradation

Apr <var data-var='date'>17</var>, <var data-var='time'>14:56</var> MDT Resolved - This incident has been resolved. Apr <var data-var='date'>17</var>, <var data-var='time'>14:35</var> MDT Monitoring - The UI is operational, metric processing is recovering. Dashboards will be caught up shortly. Apr <var data-var='date'>17</var>, <var data-var='time'>14:24</var> MDT Update - We are continuing to investigate this issue. Apr <var data-var='date'>17</var>, <var data-var='time'>14:22</var> MDT Update - We are continuing to investigate this issue. Apr <var data-var='date'>17</var>, <var data-var='time'>14:22</var> MDT Investigating - We are currently investigating this issue.

Is Scout down?

Scout status is UP

Lastest published events

Ingest Delays

Ingest Delays

Ingest delays, emergency maintenance

Scout Services, Components and Regions

Recent Scout events

Ingest Delays

Ingest Delays

Ingest delays, emergency maintenance

Ingest delays, UI latency

Ingest delays and UI latency

Database performance issues