Apr <var data-var='date'>17</var>, <var data-var='time'>12:55</var> EDT Resolved - As of 16:55 UTC, we are have resolved the degraded performance issue affecting availability of session data in us-east-1. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Apr <var data-var='date'>17</var>, <var data-var='time'>12:28</var> EDT Monitoring - As of 16:25 UTC, we have resolved the slowdown in processing session results and all sessions are being scored normally. We will continue to monitor for a further 30 minutes before closing this incident report. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Apr <var data-var='date'>17</var>, <var data-var='time'>12:21</var> EDT Identified - As of 16:20 UTC, Learnosity has increased database performance and we are processing queued sessions rapidly. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Apr <var data-var='date'>17</var>, <var data-var='time'>12:06</var> EDT Investigating - As of 16:00 UTC, we are currently experiencing a slowdown in availability of newly submitted submitted session results in the us-east-1 region. There is no data loss, all saves and submissions are being queued as designed, and the assessments stack remains operational. Delays may currently be experienced using the Data API to fetch session results, and the Reports API to display results. Other workflows that rely on the synchronous use of Data API results may also be affected. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Degraded performance affecting availability of newly submitted session results in US-East-1

Apr <var data-var='date'>16</var>, <var data-var='time'>11:23</var> EDT Resolved - As of 14:30 UTC, the Live Progress report event degradation issue in the us-east-1 region has been resolved. Additional capacity processed the event load and these systems are operating normally. Brief additional latency during this scaling may have been introduced for users of Learnosity's premium Firehose feature, but we've seen no evidence that this affected customers. This, too, was resolved once scaling was complete. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Apr <var data-var='date'>16</var>, <var data-var='time'>10:08</var> EDT Investigating - As of 13:30 UTC, we are currently experiencing slow downs affecting the Live Progress report in the us-east-1 region. Atypical use of eventbus may also be affected, such as custom reports/implementations. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Degraded performance affecting Live Progress Report in US-East-1

Feb <var data-var='date'> 4</var>, <var data-var='time'>14:15</var> EST Resolved - As of 19:12 UTC, traffic has returned to normal and everything remains stable. We have resolved the incident affecting assessments in VA (us-east-1). Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Feb <var data-var='date'> 4</var>, <var data-var='time'>13:55</var> EST Monitoring - As of 18:52 UTC, the Learnosity System Engineering team has put a fix in place. We will continue to monitor for some time to ensure a stable recovery, but all services are operational again. Learnosity Support and Systems Engineering teams will continue to actively monitor the issue, and will follow on with an update and resolution as soon as possible. Feb <var data-var='date'> 4</var>, <var data-var='time'>13:34</var> EST Identified - As of 18:30 UTC, we are currently experiencing an issue affecting assessment delivery in VA (us-east-1). Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Issue affecting assessment delivery in VA (us-east-1)

Jan <var data-var='date'>12</var>, <var data-var='time'>14:10</var> EST Resolved - As of 7:00 UTC, we've concluded our monitoring and confirmed that all new requests are behaving as expected. We are closing this record and will continue to work with affected customers directly to confirm that all recent sessions are accessible. Please reach out if you have any questions or concerns. Jan <var data-var='date'>12</var>, <var data-var='time'>12:52</var> EST Identified - As of 5:30 UTC, we've identified and corrected a load balancing configuration affecting database reads and writes for a very small number of US-only customers with consumers in the US-East-1 region. - All new sessions will behave as expected. - Read requests seeking data on sessions created prior to 23:00 UTC yesterday will also now behave correctly. - We are working on restoring sessions recently affected While all APIs continue to function normally, this update identifies service areas that may temporarily have limited access to recently saved or submitted sessions. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Jan <var data-var='date'>12</var>, <var data-var='time'>12:06</var> EST Investigating - As of 4:30 UTC, we are currently investigating an issue that may result in temporary session unavailability for a small number of US-only customers. We can confirm that no session data has been lost, and we are working to restore the affected sessions as quickly as possible. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Issue affecting availability of recent session data for a minority of customers in the US-East-1 region.

Oct <var data-var='date'>20</var>, <var data-var='time'>17:35</var> EDT Resolved - As of 21:30 UTC, all systems keep operating normally, so we are resolving the status page about the AWS outage affecting Learnosity Services in the VA (us-east-1) region. Learnosity Support and Systems Engineering teams will continue to monitor the systems closely while AWS continues to work on the issues on their side. Please reach out if you have any questions or concerns. Oct <var data-var='date'>20</var>, <var data-var='time'>16:44</var> EDT Update - As of 20:40 UTC, the error rate remains low, the infrastructure is scaled to meet demand, and the scoring queues are empty and new submissions are being processed as they come in. All services are operational and we are processing the remaining backlog of messages from the incident. AWS still lists 93 affected services and a severity of "Impacted". Learnosity Support and Systems Engineering teams are continuing to monitor the issue, but at the moment all systems are fully operational. Oct <var data-var='date'>20</var>, <var data-var='time'>15:32</var> EDT Update - As of 19:30 UTC, the error rate has fallen to 0.1% and continues to fall, we continue to steadily process the backlog of scoring messages. As of their latest update AWS has downgraded the severity of outage to "Impacted" but still list 109 services as affected. Learnosity Support and Systems Engineering teams are continuing to actively monitor the issue, and will follow on with an update and resolution as soon as possible. Oct <var data-var='date'>20</var>, <var data-var='time'>15:02</var> EDT Update - As of 20:00 UTC, the error rate has continued to drop, systems remain scaled and the backlogs of scoring messages continues to drop. AWS is still reporting 109 impacted services and a severity of "Degraded" Learnosity Support and Systems Engineering teams are continuing to monitor the issue and are manually adjusting the infrastructure where needed to ensure stability. Oct <var data-var='date'>20</var>, <var data-var='time'>14:13</var> EDT Monitoring - As of 19:10 UTC, we are beginning to see errors rates diminish and some Learnosity services are beginning to scale again. AWS is still reporting a severity of "Degraded" with problems to 104 of their services in us-east-1. Learnosity Support and Systems Engineering teams are continuing to monitor and manually intervene when possible to mitigate the effects of the outage. Oct <var data-var='date'>20</var>, <var data-var='time'>11:30</var> EDT Investigating - As of 15:25 UTC, we are seeing increasing error rates. Data API is the most affected service at the moment and we are seeing an increase in scoring delays, but we are continuing to score incoming messages to the best rate our current capacity allows. AWS is still reporting a severity of "Degraded", but now with an increased count of 81 affected services. Learnosity Support and Systems Engineering teams are continuing to monitor the issue, and will continue our mitigation efforts wherever possible. Oct <var data-var='date'>20</var>, <var data-var='time'>10:53</var> EDT Monitoring - Learnosity services in VA are continuing to be partially impacted by the major AWS outage. AWS first reported the incident at 7:11 UTC As of 14:50 UTC they are continuing to work on its full resolution, but have again set their impact level to "Degraded". The AWS Health Status dashboard where updates are being posted by the AWS team can be found here: https://health.aws.amazon.com/health/status Overall, the error rate has fallen and has stayed below what we saw earlier in the day and we have been successful in slowly scaling up the infrastructure, but we're still seeing some issues stemming from the the ongoing AWS outage as we attempt to bring online some new instances, and we have begun to see an uptick in errors. These have included errors associated with Data API, Feedback Aide, and the delivery of adaptive assessments, delays in scoring and video encoding, as well as other smaller spikes across different systems. While the overall error rate remains low, Learnosity Support and Systems Engineering teams are actively monitoring the outage and manually intervening where possible to mitigate the issues we're facing. In spite of our best efforts it is likely that the APIs may not be fully operational in VA until AWS's service is fully restored to normal operation, but we are taking every step within our power to try to minimize the disruption to our services.

AWS outage continuing to have some impact on Learnosity services in VA (us-east-1)

Oct <var data-var='date'>20</var>, <var data-var='time'>07:23</var> EDT Resolved - As of 11:20 UTC, AWS services appear to have been stable for approximately an hour. Learnosity APIs--including services affected by the AWS outage (such as CloudFront)--appear to be fully operational. At this time we are labeling this issue resolved. Learnosity Support and Systems Engineering teams will continue to investigate impact from this incident and follow up with a post mortem once we hear from AWS. Please reach out if you have any questions or concerns. Oct <var data-var='date'>20</var>, <var data-var='time'>06:11</var> EDT Monitoring - AWS is going through late-stage recovery of a major outage in US-East-1 bringing down most of their services. Learnosity APIs were stable and unaffected in isolation, but some impact cascaded to Learnosity (such as availability of assets from CloudFront). The full extent of the incident is unknown at this time. NOTE: Up until the time of this writing (10:05 UTC), this included impact to sign-in services that prevented us from accessing our StatusPage. We immediately posted banner notifications in our Support Portal, and those banners will remain visible until we've been able to verify that service has been restored for sufficient time. Incident History: As of 7:00 UTC, AWS experienced a major outage. While Learnosity APIs were fully operational, some Learnosity services use AWS and were affected. These included access to our Support ticketing system (the Learnosity-hosted Help guides were unaffected), and availability of CloudFront-hosted assets, but other systems may have been impacted as well. As of 10:00 UTC, AWS has restored services and access to Learnosity-adjacent features is becoming available relatively quickly. Learnosity Support will monitor AWS for a further 30 minutes before calling this issue resolved. We will follow on with an update and resolution at that time, or sooner.

AWS Outage Impacted some Learnosity Services

Oct <var data-var='date'> 2</var>, <var data-var='time'>18:43</var> EDT Resolved - As of 22:42 UTC, all Learnosity APIs are continuing to function as expected in all regions, and we are declaring this issue resolved. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Oct <var data-var='date'> 2</var>, <var data-var='time'>18:14</var> EDT Monitoring - As of 22:12 UTC, Desmos services have been restored and Learnosity APIs are initializing without issue. Desmos and Learnosity Support and Systems Engineering teams will continue to monitor for a further 30 minutes before declaring this issue resolved. We will follow on with an update and resolution as soon as possible. Oct <var data-var='date'> 2</var>, <var data-var='time'>17:34</var> EDT Investigating - As of 21:15 UTC, Learnosity APIs are being impacted by a temporary outage from our Math partner, Desmos. Authoring or Editing content with Desmos questions or features, or initializing assessments that contain Desmos questions or features, can prevent the Author API or Items API from initializing. Specific reports that display question content, such as the Session Detail by Item report, can also fail to initialize Learnosity APIs are not otherwise impacted, so content that does not include Desmos features are not affected. Until we learn more, we will list APIs that may experience knock-on effects from this temporary Desmos outage. However, as most customers will not experience any issues, we will not list these APIs as down. Desmos and Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Partner Service Outage May Disrupt Learnosity API Initialization in All Regions

Aug <var data-var='date'>19</var>, <var data-var='time'>17:31</var> EDT Resolved - As of 21:30 UTC, we are have resolved the issue affecting availability of session data in US-East-1. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Aug <var data-var='date'>19</var>, <var data-var='time'>17:05</var> EDT Monitoring - As of 21:00 UTC, all sessions have been cleared from the scoring backlog queue and all submissions are being processed normally. Learnosity Support and Systems Engineering teams will monitor this situation for a further 30 minutes before calling it resolved. Aug <var data-var='date'>19</var>, <var data-var='time'>16:35</var> EDT Update - As of 20:30 UTC, the scoring queue backlog is almost empty and new sessions will soon be processed without delay. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Aug <var data-var='date'>19</var>, <var data-var='time'>15:56</var> EDT Update - As of 19:30 UTC, we are now processing queued sessions rapidly and more than half of the backlog has already cleared.. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Aug <var data-var='date'>19</var>, <var data-var='time'>14:42</var> EDT Update - As of 18:30 UTC, initial remediation efforts have tripled the rate of queued session processing and we are continuing to work toward a full resolution. Access to recently submitted session results via the Data API and Reports API remains the only affected part of the Learnosity ecosystem. New submissions continue to be safely queued for scoring while the degraded performance remains. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Aug <var data-var='date'>19</var>, <var data-var='time'>13:34</var> EDT Identified - (Note: We're correcting cited UTC times to the 24 hr format and will include both forms in this update only.) As of 5:30pm/17:30 UTC, we've identified a possible contributing cause for the degraded performance in our analytics stack. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Aug <var data-var='date'>19</var>, <var data-var='time'>12:35</var> EDT Update - As of 4:30 UTC, we are continuing to investigate degraded performance in our Data and Reports APIs. Only availability of recently submitted session data is affected. Historic session data, as well as all other API stacks, remain unaffected. New submissions are persisting correctly with no data loss and these submission are being queued for scoring. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Aug <var data-var='date'>19</var>, <var data-var='time'>12:11</var> EDT Investigating - As of 3:40 UTC, we are experiencing degraded performance in our analytics stack affecting the availability of session data in US-East-1 for a subset of customers. This is affecting the Reports API and Data API. Neither authoring nor assessment stacks are affected, and there is no data loss. Submitted sessions are queueing for processing. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Issue affecting availability of recently submitted session analytics in US-East-1

Aug <var data-var='date'> 8</var>, <var data-var='time'>10:19</var> EDT Resolved - As of 14:10 UTC, we've found no negative behavior in the manual update processes and attribute the short burst of jobs to uncharacteristic testing of the new manual grading workflow. The jobs queue functioned as expected, ensuring no loss of requests, and the jobs were processed in a timely fashion. Learnosity will continue to monitor this activity for a further 60 minutes, but we are calling this resolved at this time. We apologize for the unwarranted alert. Please reach out if you have any questions or concerns. Aug <var data-var='date'> 8</var>, <var data-var='time'>09:57</var> EDT Update - As of 13:55 UTC, all manual update jobs have been executed and the jobs queue is empty. Requests to manually update session response scores, session statuses, and session metadata should execute as expected. We are still investigating possible reasons for the blip. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Aug <var data-var='date'> 8</var>, <var data-var='time'>09:55</var> EDT Investigating - As of 13:40 UTC, we began investigating a possible issue affecting manual update jobs to session response scores, sessions statuses, and session metadata in the US-East-1 (VA) region. We saw a brief buildup in the queue for these jobs, the queue processed as expected, and is now empty. However, we're investigating further, out of an abundance of caution, to see if there are any unexpected causes for the short queue buildup. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Possible Issue affecting manual updates to session data in US-East-1 (VA) region

Mar <var data-var='date'> 3</var>, <var data-var='time'>10:54</var> EST Resolved - As of 15:45 UTC, after a further 30 minutes of monitoring, we are resolving this issue. All services remain fully operational. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required. Please reach out if you have any questions or concerns. Mar <var data-var='date'> 3</var>, <var data-var='time'>10:20</var> EST Monitoring - As of 15:15 UTC, all services have been restored. All queued messages have been processed and session information should now be available. Users of the Learnosity Firehose service will still see slight delays as messages are sent out for recently processed events. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Mar <var data-var='date'> 3</var>, <var data-var='time'>10:06</var> EST Update - As of 15:00 UTC, we are continuing to investigate the issue. Data and Reports API are still impacted. Assessment and Authoring APIs remain unaffected. Assessment submissions are being queued and will be processed as the system works through any backlog queue. Note that if the Data API or Reports API is used in a preliminary/synchronous assessment delivery workflow, this will likely have a knock-on affect when initializing assessments. Initializing any assessment API directly is unaffected at present. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Mar <var data-var='date'> 3</var>, <var data-var='time'>09:00</var> EST Update - As of 14:00 UTC, we are still investigating an issue affecting the availability of session information in US-East-1. Both Data API and Reports API are experiencing extended delays and intermittent failures in returning session results. Authoring and Assessment APIs appear to be unaffected at this time and all session submissions are being successfully queued. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Mar <var data-var='date'> 3</var>, <var data-var='time'>08:48</var> EST Investigating - As of 13:45 UTC, we are experiencing delays in availability of session information in US-East-1 Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Issue affecting availability of session information in US-East-1

Jan <var data-var='date'> 9</var>, <var data-var='time'>09:50</var> EST Resolved - As of 14:50 UTC, after a further 60 minutes of issue-free monitoring, we are resolving this issue affecting availability of session information via our Analytics stack. Reports API and Data API are fully operational. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Jan <var data-var='date'> 9</var>, <var data-var='time'>09:13</var> EST Monitoring - As of 14:10 UTC, Learnosity has restored full service for all new requests to all Analytics APIs. We are monitoring stability and analyzing all queued requests. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Jan <var data-var='date'> 9</var>, <var data-var='time'>09:02</var> EST Update - As of 14:00 UTC, Learnosity is working on identifying the cause of degraded performance in our Analytics APIs, Reports and Data. Authoring and Assessment stacks remain unaffected Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Jan <var data-var='date'> 9</var>, <var data-var='time'>08:33</var> EST Investigating - As of 13:30 UTC, We are currently investigating a possible performance degradation affecting Reports API and Data API results in US-East-1. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Analytics APIs Experiencing Degraded Performance in US-East-1

Oct <var data-var='date'> 1</var>, <var data-var='time'>12:07</var> EDT Resolved - As of 16:05 UTC, We are have resolved the issue affecting live progress reports in VA (us-east-1). As the recovery remains stable, we are now resolving the incident that was affecting live progress reports. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required. Please reach out if you have any questions or concerns. Oct <var data-var='date'> 1</var>, <var data-var='time'>11:40</var> EDT Monitoring - As of 15:35 UTC, the Systems Engineering team has deployed a fix, the errors have stopped, and currently all systems are functional. We will continue to monitor for a time before resolving the incident but we are currently seeing no errors. We will follow on with an update and resolution as soon as possible. Oct <var data-var='date'> 1</var>, <var data-var='time'>11:25</var> EDT Identified - As of 15:20 UTC, the Systems Engineering team has identified the cause for the slower than normal delivery of events and is proceeding with mitigation steps. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Oct <var data-var='date'> 1</var>, <var data-var='time'>11:09</var> EDT Investigating - As of 15:00 UTC, we are currently investigating an issue affecting the event delivery that powers the live progress report (live-activitystatus-by-user) in VA (us-east-1). Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Investigating issue affecting live progress report / proctoring in VA

Sep <var data-var='date'>13</var>, <var data-var='time'>12:03</var> EDT Resolved - As of 16:00 UTC, after monitoring and confirming the stable state of the fixes put into place, we have resolved the Events API events delivery issue affecting proctored assessments in us-east-1 (VA). Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required. Please reach out if you have any questions or concerns. Sep <var data-var='date'>13</var>, <var data-var='time'>11:34</var> EDT Monitoring - As of 15:30 UTC, events are flowing normally once again and all systems are operational. We will continue to monitor for some time before we resolve the issue fully to ensure we have a stable recovery. Sep <var data-var='date'>13</var>, <var data-var='time'>11:09</var> EDT Update - As of 15:00 UTC, the Learnosity Systems Engineering team has identified the issue preventing the assessment events from being delivered and are working on the implementation of remediation steps. Learnosity Support and Systems Engineering teams are continuing to work on the issue, and will follow on with an update and resolution as soon as possible. Sep <var data-var='date'>13</var>, <var data-var='time'>11:06</var> EDT Identified - As of 15:00 UTC, the Learnosity Systems Engineering team has identified the issue preventing the assessment events from being delivered and are working on the implementation of remediation steps. Learnosity Support and Systems Engineering teams are continuing to work on the issue, and will follow on with an update and resolution as soon as possible. Sep <var data-var='date'>13</var>, <var data-var='time'>10:35</var> EDT Investigating - As of 14:30 UTC, We are investigating a potential issue affecting the delivery of some Events API events (used in the live proctoring / live-activitystatus-by-user report) in us-east-1 (VA). Assessments are otherwise proceeding normally and can be saved and submitted without any issue. This issue if confirmed would only affect the delivery of Events API events. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Investigating potential issue affecting live proctoring in us-east-1 (VA)

May <var data-var='date'>22</var>, <var data-var='time'>15:42</var> EDT Resolved - As of 19:40 UTC, we believe we've identified content to be the root cause of this tiny number of sessions not passing through scoring and triggering a Firehose event. We've reached out to customer proactively and will help address this issue directly. We are now calling this issue resolved and apologize for any distractions. Please reach out if you have any questions or concerns. May <var data-var='date'>22</var>, <var data-var='time'>15:01</var> EDT Monitoring - As of 19:00 UTC, we have been unable to find any systemic problems, while the number of potentially impacted sessions remains at one hundredth of one percent ( 0.01%) of total submissions over the period investigated, likely due to a content issue. Learnosity Support and Systems Engineering teams will continue to monitor for 30 minutes and expect to close this investigation at that time. May <var data-var='date'>22</var>, <var data-var='time'>14:47</var> EDT Update - As of 18:45 UTC, Learnosity Support and Systems Engineering teams are continuing to investigate an unconfirmed issue, examining less than 500 potentially affected sessions out of 4.93 million submitted over the same period. We will follow on with an update as soon as possible. May <var data-var='date'>22</var>, <var data-var='time'>14:33</var> EDT Update - As of 18:30 UTC, Learnosity Support and Systems Engineering teams are continuing to investigate this unconfirmed issue and will follow on with an update as soon as possible. May <var data-var='date'>22</var>, <var data-var='time'>14:27</var> EDT Investigating - As of 18:10 UTC, we are trying to determine if an issue exists causing a very small number of adaptive sessions to not trigger a Firehose event after scoring in US-East-1. This is just an internal investigation at present, without confirmation, and we have no evidence that any impact outside the described implementation is affected. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Possible issue affecting small number of adaptive sessions not triggering Firehose updates in US-East-1

Apr <var data-var='date'>30</var>, <var data-var='time'>11:24</var> EDT Resolved - As of 15:00 UTC, we are closing this incident as resolved. The availability and performance of the Data API and downstream systems in US-East-1 has been restored and operating correctly for 30 minutes. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Apr <var data-var='date'>30</var>, <var data-var='time'>10:54</var> EDT Monitoring - As of 14:30 UTC, we have addressed an issue primarily affecting Data API use. Error rate has reduced to zero, all newly submitted sessions are scoring immediately, and the backlog of sessions awaiting scoring is cleared. The Firehose service queue is clearing quickly dispatching information about recently scored sessions. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Apr <var data-var='date'>30</var>, <var data-var='time'>09:49</var> EDT Update - As of 13:45 UTC, we are investigating an issue with the availability of authoring and assessment content in the US-East-1 region. This issue is also impacting analytics, with degraded performance of Reports and select Data API endpoints that require content, such as the Session Detail by Item report and Data API item bank and scoring endpoints. We will continue to update this incident as we learn more. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Apr <var data-var='date'>30</var>, <var data-var='time'>09:29</var> EDT Investigating - We are currently experiencing an issue affecting the availability of newly authored and edited content. This is also having a ripple effect on assessments dynamically retrieving content, such as adaptive and on-the-fly assessments. We are investigating and will update this record as soon as we learn more.

Issue affecting authoring and specific kinds of assessments in U.S.-East-1

Apr <var data-var='date'>25</var>, <var data-var='time'>18:00</var> EDT Resolved - As of 21:56 UTC, all jobs continue to be processed at a normal rate and all services are operational. We are marking the issue affecting the processing of item pool and offline package jobs in US-West-2 as resolved. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required. Please reach out if you have any questions or concerns. Apr <var data-var='date'>25</var>, <var data-var='time'>17:30</var> EDT Monitoring - As of 21:26 UTC, the jobs processing performance has returned to normal and all job queues are empty. Everything is currently performing as expected, we will continue to monitor for a time to ensure a full recovery before fully resolving the incident. Apr <var data-var='date'>25</var>, <var data-var='time'>17:19</var> EDT Identified - As of 21:15 UTC, the Learnosity Systems Engineering team has identified the issue and is making adjustments to restore normal jobs performance. Learnosity Support and Systems Engineering teams are continuing to work on the issue, and will follow on with an update and resolution as soon as possible. Apr <var data-var='date'>25</var>, <var data-var='time'>16:43</var> EDT Investigating - As of 20:30 UTC, we are investigating a possible issue affecting the processing of item pool and offline package jobs in the US-West-2 region and causing slower than normal processing times. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Investigating possible degraded performance affecting some asynchronous jobs in the US-West-2 Region

Jan <var data-var='date'>26</var>, <var data-var='time'>12:14</var> EST Resolved - As of 17:00 UTC, we've identified the problem affecting tag hierarchies and it is not an infrastructure/services issue. We've determined that an application-level bug limited to saving a newly edited hierarchy is the root cause and have raised a ticket for processing. All other API and backend functionality, including the use of existing tag hierarchies, remains operational. We are closing this incident and moving the focus to the software team. If you experience any issues with saving hierarchies please don't hesitate to raise a ticket with the Learnosity Support team and we will provide additional updates. Thank you. Jan <var data-var='date'>26</var>, <var data-var='time'>11:02</var> EST Update - As of 16:00 UTC, we are continuing to investigate an issue preventing updates to Tag Hierarchies. This issue is limited to the creation and editing of tag hierarchies. Only a small subset of reports that make use of this feature may be affected, and only when a change to a tag hierarchy is required. All other APIs, including reports that don't rely on hierarchy updates, remain operational. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Jan <var data-var='date'>26</var>, <var data-var='time'>10:00</var> EST Investigating - As of 14:30 UTC, we are investigating an issue preventing tag hierarchies from being created or updated in the US-East-1 Region. This only appears to affect new changes to tag hierarchies, so this should only impact the use of select reports that rely on this feature, and only when a newly created or updated hierarchy is required. Routine authoring and assessments are unaffected, as are reports and Data API use that don't rely on newly edited hierarchies. We're including tag-related reports in this update for visibility, but the Reports API remains operational and only a small subset of reports use tag heirarchies. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Issue affecting Tag Hierarchies in the US-East-1 region

Dec <var data-var='date'>13</var>, <var data-var='time'>14:42</var> EST Resolved - As of 19:40 UTC, we've concluded one hour of active monitoring without additional issues. We will continue to keep an eye on everything but are now ready to call this incident resolved. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Dec <var data-var='date'>13</var>, <var data-var='time'>13:36</var> EST Monitoring - As of 18:36 UTC, we've now recovered fully and all stacks are operational. We will continue to monitor for a period before resolving the incident. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Dec <var data-var='date'>13</var>, <var data-var='time'>13:25</var> EST Update - As of 18:20 UTC, we've verified a cause of the current incident and recovery is already partially complete. We have restored part of the assessment stack and are working on full restoration now, with the remaining stacks to follow. We will provide additional updates soon, including a possible ETA as soon as a reasonably accurate estimate has been determined. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Dec <var data-var='date'>13</var>, <var data-var='time'>12:16</var> EST Identified - We've now identified a possible cause for the current incident and are working to resolve the issue. We will add additional information as soon as it becomes available. We've also added EMEA authoring and APAC authoring to affected regions for users of the hosted author site, which operates in the AMER region. Learnosity Support and Systems Engineering teams are continuing to actively investigate the current issue, and will follow on with an update and resolution as soon as possible. Dec <var data-var='date'>13</var>, <var data-var='time'>10:40</var> EST Update - Correction: The authoring and analytics stacks are also begin impacted intermittently in workflows where the assessment stack is in play. (E.g. editing views where the assessment stack is used for previews.) Errors in the AMER region continue to reduce. 502 errors have cleared and 504 timeout errors are now being investigated. Elevating classifications to include all stacks and partial outage. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Dec <var data-var='date'>13</var>, <var data-var='time'>10:06</var> EST Investigating - As of 14:15 UTC, the assessment stack returned to normal operating metrics in the EMEA and APAC regions. As of 14:30 and 14:45 UTC, the NA region error count dropped by approximately 15% each period, and error counts are continuing to reduce as of 15:00 UTC. Intermittent errors continue to occur in the assessment stack in AMER. Authoring and Analytics APIs remain unaffected. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Dec <var data-var='date'>13</var>, <var data-var='time'>09:37</var> EST Identified - As of 14:15 UTC, we've identified that the AWS Auto Scaling Groups failed to create additional EC2 instances causing an elevated number of 502 errors in the assessment stack. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Dec <var data-var='date'>13</var>, <var data-var='time'>09:06</var> EST Investigating - As of 14:00 UTC, We are seeing intermittent issues with loading demo assessment content and are investigating out of an abundance of caution. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Issues loading assessment content in AMER region

Dec <var data-var='date'> 5</var>, <var data-var='time'>14:35</var> EST Resolved - As of 17:55 UTC, we identified an adaptive Redis cache that was intermittently refusing a small percentage of connections. Self-hosted adaptive makes extensive use of this cache due to the need to store more state information supplied by the self-hosted client. We replaced the cache database, routed all traffic to the new instance, and saw an immediate, dramatic reduction of refused connections. We've monitored for approximately 90 minutes and all activity indicates normal operation of all APIs. We are resolving this issue and will continue to monitor customer comms. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Dec <var data-var='date'> 5</var>, <var data-var='time'>12:29</var> EST Investigating - As of 17:25 UTC, we are investigating a possible issue affecting Self-Hosted Adaptive testing in the US-East-1 region. Due to the very small number of customers affected, we've not yet confirmed if this is something related to Learnosity APIs or the self-hosted algorithm portion of the solution. Most customers using this feature appear to be unaffected, but we're raising this alert out of an abundance of caution. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Investigating possible degraded performance affecting Self-Hosted Adaptive in the US-East-1 Region

Dec <var data-var='date'> 4</var>, <var data-var='time'>11:47</var> EST Resolved - As of 16:45 UTC, we are have resolved the issue affecting scoring delays in the US-East-1 region. All sessions are scoring as expected, with no backlog or delays. No data was lost during this incident, the authoring and assessment stacks were unaffected, and analytics have been restored. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Dec <var data-var='date'> 4</var>, <var data-var='time'>11:11</var> EST Monitoring - As of 16:10 UTC, incoming submissions are now being scored with expected turnaround, and the queue of previously submitted sessions awaiting scoring has been cleared. Session results, access to the Data API sessions/responses endpoint, and analytics via the Reports API for affected sessions, should all function normally. All APIs are fully operational. Learnosity Support and Systems Engineering teams will continue to actively monitor the issue, and will follow on with an update and resolution as soon as possible. Dec <var data-var='date'> 4</var>, <var data-var='time'>10:55</var> EST Update - As of 15:40 UTC, we are addressing an issue with scoring delays in the US-East-1 region. Session results, access to the Data API sessions/responses endpoint, and analytics via the Reports API for affected sessions, are showing delays. Submissions are continuing to be accepted with no data loss, and we are processing the backlog of submissions awaiting scoring. No other APIs are affected. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Dec <var data-var='date'> 4</var>, <var data-var='time'>09:27</var> EST Identified - As of 14:20 UTC, We are currently experiencing an issue affecting learner session data retrieval in VA (us-east-1). Learnosity Support and Systems Engineering teams have identified the issue and are working to address it, and will follow on with an update and resolution as soon as possible.

Issue affecting session data retrieval in the VA region (us-east-1)

Nov <var data-var='date'>24</var>, <var data-var='time'>09:58</var> EST Resolved - As of 14:53 UTC, We are have resolved the delay affecting authoring in all regions. Learnosity Support and Systems Engineering teams will follow up with a post-mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Nov <var data-var='date'>24</var>, <var data-var='time'>09:22</var> EST Monitoring - As of 14:00 UTC, normal authoring performance has been restored in the remaining regions and all systems are now operating as expected across all regions. Clients should no longer be experiencing any delays in being able to access newly created or edited content in any Learnosity region. We will continue to monitor for some time to ensure that all systems continue to operate normally before fully resolving this issue. Nov <var data-var='date'>24</var>, <var data-var='time'>08:35</var> EST Update - As of 13:20 UTC, the delay in availability of newly authored or edited content has resolved for the AU region. Our engineers continue to work on the resolution for the rest of the regions and we will provide an update as soon as those have also recovered. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible. Nov <var data-var='date'>24</var>, <var data-var='time'>07:28</var> EST Identified - As of 07:33 UTC, We are currently experiencing a delay in the availability of newly authored or edited of content in all regions. New or edited content may be delayed in appearing in the authoring interface and in being available to be used in assessments. Existing content is unaffected and will be available to assessments as usual. Learnosity Support and Systems Engineering teams have identified the cause of the issue and are actively working on it, and will follow on with an update and resolution as soon as possible.

Issue affecting authoring and delivery of new content in all regions

Oct <var data-var='date'> 2</var>, <var data-var='time'>13:14</var> EDT Resolved - As of 16:55 UTC, after two hours of monitoring, all scoring queues remain empty as sessions are being scored promptly. We are resolving this issue. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns. Oct <var data-var='date'> 2</var>, <var data-var='time'>10:59</var> EDT Update - As of 14:55 UTC, all queued sessions have completed scoring and new sessions are moving through the scoring process smoothly. Analytical data and Firehose messages should be available for all sessions. No authoring or assessment flows were affected, all submissions were successfully saved without data loss, and the small subset of customers in the VA region that were affected should again be fully operational. Learnosity Support and Systems Engineering teams will continuing to monitor the issue, and will follow on with an update and resolution as soon as possible. Oct <var data-var='date'> 2</var>, <var data-var='time'>10:49</var> EDT Monitoring - As of 14:30 UTC, we identified and corrected the issue. Our engineers are now monitoring the queue which is diminishing quickly. Learnosity Support and Systems Engineering teams are monitoring the issue, and will follow on with an update and resolution as soon as possible. Oct <var data-var='date'> 2</var>, <var data-var='time'>10:06</var> EDT Investigating - As of 14:00 UTC, we are investigating an issue with asynchronous scoring delays affecting our US-East-1 region. Note: Submissions are not affected, resulting in no data loss, and no other APIs are affected. Sessions are being successfully stored and submitted for scoring, going into a temporary backlog queue. This slows down availability of analytics, including notification delays in the otherwise fully functional Firehose service. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Scoring delays impacting a small number of clients in the US-East-1 Region

Sep <var data-var='date'>14</var>, <var data-var='time'>13:45</var> EDT Resolved - As of 17:45 UTC, we are no longer experiencing issues with Support tickets loading, and are calling this third-party issue resolved. Sep <var data-var='date'>14</var>, <var data-var='time'>10:38</var> EDT Investigating - As of 14:30 UTC, we are currently seeing a partial outage in our third-party ticketing system. Zendesk status doesn't currently reflect an issue, but the ability to load or create support tickets may be affected. Learnosity Support is actively monitoring the issue, and will follow on with an update and resolution as soon as possible.

Investigating unresponsive support ticketing system

Sep <var data-var='date'>11</var>, <var data-var='time'>15:26</var> EDT Resolved - As of 19:20 UTC, our third-party support vendor states that no issues are affecting Learnosity Support. (One incident remains active but is not thought to affect the Learnosity Support ticketing system.) We are resolving this incident but will reopen if we are affected. Please reach out if you have any questions or concerns. Sep <var data-var='date'>11</var>, <var data-var='time'>15:06</var> EDT Monitoring - As of 19:00 UTC, our ticketing system vendor is reporting an ongoing incident affecting performance of the Learnosity Help ticketing system. We will continue to monitor and report progress as we become aware of changes.

Support ticketing system vendor experiencing degraded performance

Aug <var data-var='date'>24</var>, <var data-var='time'>01:01</var> EDT Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>23</var>, <var data-var='time'>19:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>16</var>, <var data-var='time'>15:12</var> EDT Scheduled - Between August 23, 2023, 19:00 UTC and August 24, 2023, 01:00 UTC, Zendesk will be conducting maintenance that affects our support system instance. Per Zendesk: >[Customers] will experience a maximum of two separate service disruptions totalling 5-10 minutes at some time within [the scheduled] window.

Is Learnosity down?

Learnosity status is UP

Lastest published events

Degraded performance affecting availability of newly submitted session results in US-East-1

Degraded performance affecting Live Progress Report in US-East-1

Issue affecting assessment delivery in VA (us-east-1)

Learnosity Services, Components and Regions

Recent Learnosity events

Degraded performance affecting availability of newly submitted session results in US-East-1

Degraded performance affecting Live Progress Report in US-East-1

Issue affecting assessment delivery in VA (us-east-1)

Issue affecting availability of recent session data for a minority of customers in the US-East-1 region.

AWS outage continuing to have some impact on Learnosity services in VA (us-east-1)

AWS Outage Impacted some Learnosity Services

Get your FREE account now,
5 monitors included!

Is Learnosity down?

Learnosity status is UP

Lastest published events

Degraded performance affecting availability of newly submitted session results in US-East-1

Degraded performance affecting Live Progress Report in US-East-1

Issue affecting assessment delivery in VA (us-east-1)

Learnosity Services, Components and Regions

Recent Learnosity events

Degraded performance affecting availability of newly submitted session results in US-East-1

Degraded performance affecting Live Progress Report in US-East-1

Issue affecting assessment delivery in VA (us-east-1)

Issue affecting availability of recent session data for a minority of customers in the US-East-1 region.

AWS outage continuing to have some impact on Learnosity services in VA (us-east-1)

AWS Outage Impacted some Learnosity Services

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!