Aug <var data-var='date'>21</var>, <var data-var='time'>21:39</var> EDT Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>21</var>, <var data-var='time'>21:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>20</var>, <var data-var='time'>17:54</var> EDT Scheduled - We will be undergoing scheduled maintenance during this time.

System maintenance

Aug <var data-var='date'>18</var>, <var data-var='time'>21:30</var> EDT Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>18</var>, <var data-var='time'>21:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Aug <var data-var='date'>17</var>, <var data-var='time'>17:28</var> EDT Scheduled - We will be undergoing scheduled maintenance during this time. This is expected to take no more than 30 minutes. During this time, writes, including updates to jobs or persons or generating new reports, may fail with a "Service Unavailable" error message. Users will be able to continue to navigate the product and retrieve information during this time.

Jun <var data-var='date'>17</var>, <var data-var='time'>14:21</var> EDT Resolved - This incident has been resolved. Jun <var data-var='date'>17</var>, <var data-var='time'>13:38</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>17</var>, <var data-var='time'>12:55</var> EDT Identified - An issue deserializing data from our database is preventing ChartHop from loading certain data.

ChartHop is not loading

Mar <var data-var='date'>26</var>, <var data-var='time'>19:20</var> EDT Resolved - This incident has been resolved. Mar <var data-var='date'>26</var>, <var data-var='time'>18:35</var> EDT Investigating - We are currently investigating this issue.

ChartHop API is not responding

Feb <var data-var='date'>13</var>, <var data-var='time'>13:24</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>13</var>, <var data-var='time'>12:38</var> EST Monitoring - ChartHop leverages MongoDB as a primary data store. MongoDB experienced a major service outage impacting data stores in AWS. This impacts ChartHop and is caused our application to fail. The issue has been mitigated and we are monitoring.

MongoDB incident impacting ChartHop

Feb <var data-var='date'> 9</var>, <var data-var='time'>13:54</var> EST Resolved - The system is operating as expected and error rates have returned to normal. Feb <var data-var='date'> 9</var>, <var data-var='time'>13:44</var> EST Monitoring - We have identified an issue with routing between our User Interface and API servers that was sending a portion of API traffic to the wrong servers. The routing issue has been addressed and service should be restored for all users. Feb <var data-var='date'> 9</var>, <var data-var='time'>12:59</var> EST Investigating - We are seeing an increased rate of errors specifically coming from the ChartHop user interface. This is impacting users ability to navigate and use the product. API servers and Apps/integrations are not seeing increased error rates overall, and impacts are limited to UI access.

High error rates on UI

Jun <var data-var='date'> 4</var>, <var data-var='time'>12:41</var> EDT Resolved - This incident has been resolved. Jun <var data-var='date'> 4</var>, <var data-var='time'>12:13</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 4</var>, <var data-var='time'>11:30</var> EDT Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'> 4</var>, <var data-var='time'>10:56</var> EDT Investigating - Users report that ChartHop is slow to load, sitting on the "loading bunny" screen for an extended time.

ChartHop is slow to load

Mar <var data-var='date'>28</var>, <var data-var='time'>16:33</var> EDT Resolved - This incident has been resolved. Mar <var data-var='date'>28</var>, <var data-var='time'>14:21</var> EDT Monitoring - The update has been fully deployed, and we are monitoring for any remaining impacts. Mar <var data-var='date'>28</var>, <var data-var='time'>13:40</var> EDT Update - An update is being deployed to address the issue and resolve service for the remaining impacted users. Mar <var data-var='date'>28</var>, <var data-var='time'>12:26</var> EDT Identified - Some ChartHop users are unable to use the ChartHop application at this time and getting either a blank page or a failure to load. We have identified the issue and working to restore service.

ChartHop app is not loading for some users

Feb <var data-var='date'>10</var>, <var data-var='time'>12:04</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>10</var>, <var data-var='time'>11:48</var> EST Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>10</var>, <var data-var='time'>11:24</var> EST Identified - We have been made aware that some customers are unable to open the ChartHop user interface. We have identified the issue and a fix is being deploy. ChartHOP API and integrations is unaffected.

User interface unavailable

Apr <var data-var='date'>26</var>, <var data-var='time'>18:14</var> EDT Resolved - This incident has been resolved. Apr <var data-var='date'>26</var>, <var data-var='time'>17:02</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>26</var>, <var data-var='time'>16:36</var> EDT Investigating - We are currently investigating a performance degradation for some dashboards and reports. There are no known impacts outside of dashboards and reports.

Reports slow

Apr <var data-var='date'>19</var>, <var data-var='time'>15:44</var> EDT Resolved - This incident has been resolved. Apr <var data-var='date'>19</var>, <var data-var='time'>11:50</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>19</var>, <var data-var='time'>11:49</var> EDT Update - We are continuing to work on a fix for this issue. Apr <var data-var='date'>19</var>, <var data-var='time'>10:49</var> EDT Update - We are continuing to work on a fix for this issue. Apr <var data-var='date'>19</var>, <var data-var='time'>09:54</var> EDT Update - We are continuing to work on a fix for this issue. Apr <var data-var='date'>19</var>, <var data-var='time'>09:42</var> EDT Identified - We're experiencing intermittent but severe performance degradation across the platform.

Severe Performance Degradation

Apr <var data-var='date'> 9</var>, <var data-var='time'>15:54</var> EDT Resolved - This incident has been resolved. Apr <var data-var='date'> 9</var>, <var data-var='time'>15:53</var> EDT Update - We are continuing to monitor for any further issues. Apr <var data-var='date'> 9</var>, <var data-var='time'>15:19</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'> 9</var>, <var data-var='time'>14:22</var> EDT Identified - The issue has been identified and a fix is being implemented. Apr <var data-var='date'> 9</var>, <var data-var='time'>14:22</var> EDT Investigating - ChartHop Engineering is investigating degraded performance in the ChartHop API that is impacting the user interface. A solution has already been identified and is in process of testing and deployment to our fleet.

Degraded performance

Sep <var data-var='date'>28</var>, <var data-var='time'>12:30</var> EDT Resolved - This incident has been resolved. Sep <var data-var='date'>28</var>, <var data-var='time'>11:59</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'>28</var>, <var data-var='time'>11:27</var> EDT Identified - We've identified some additional issues impacting the preview and testing of Compensation Reviews. Additional work is in progress. Sep <var data-var='date'>28</var>, <var data-var='time'>11:26</var> EDT Update - We are continuing to monitor for any further issues. Sep <var data-var='date'>28</var>, <var data-var='time'>10:57</var> EDT Monitoring - A fix has been deployed and we are monitoring the situation. Sep <var data-var='date'>28</var>, <var data-var='time'>10:06</var> EDT Identified - The Compensation Review API endpoints and UI elements are currently erroring out. The team has investigated and identified the cause and is working to deploy a fix.

Compensation Reviews are not loading

Jul <var data-var='date'>18</var>, <var data-var='time'>16:22</var> EDT Resolved - No new CloudFront errors have been reported. Jul <var data-var='date'>18</var>, <var data-var='time'>13:19</var> EDT Monitoring - CloudFront errors appear to have stabilized and we are not seeing continued errors accessing the ChartHop Dashboard at this time. Jul <var data-var='date'>18</var>, <var data-var='time'>13:07</var> EDT Identified - The ChartHop Dashboard is occasionally failing to load with various error messages from AWS' CloudFront service. This appears to be an AWS issue as being reported by multiple services on Twitter and other sources. ChartHop has reached out to AWS for assistance and status.

ChartHop Dashboard occasional failures

May <var data-var='date'> 2</var>, <var data-var='time'>12:45</var> EDT Resolved - This incident has been resolved. May <var data-var='date'> 2</var>, <var data-var='time'>12:40</var> EDT Monitoring - Reports are now working again. May <var data-var='date'> 2</var>, <var data-var='time'>12:21</var> EDT Identified - We are currently experiencing slowness when generating reports. We have identified the issue and are working to resolve this.

Reports service experiencing delays

Mar <var data-var='date'>20</var>, <var data-var='time'>10:35</var> EDT Resolved - We have resolved the issue, and Bands pages are now rendering. Mar <var data-var='date'>20</var>, <var data-var='time'>09:57</var> EDT Identified - We have identified the issue, and a fix is being rolled out. Mar <var data-var='date'>20</var>, <var data-var='time'>09:53</var> EDT Update - We are continuing to investigate this issue. Mar <var data-var='date'>20</var>, <var data-var='time'>09:53</var> EDT Investigating - Currently, our Band pages are not loading. The team is investigating and will update you shortly.

Band pages are not loading

Mar <var data-var='date'> 2</var>, <var data-var='time'>12:39</var> EST Resolved - This incident has been resolved. Mar <var data-var='date'> 2</var>, <var data-var='time'>11:39</var> EST Update - We are continuing to monitor for any further issues. Mar <var data-var='date'> 2</var>, <var data-var='time'>11:39</var> EST Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'> 2</var>, <var data-var='time'>11:14</var> EST Identified - Engineering has identified the root cause of the performance degradation and a fix is in process. Mar <var data-var='date'> 2</var>, <var data-var='time'>11:10</var> EST Investigating - We are seeing degraded performance on our API servers, leading to occasional errors seen in the User Interface. Engineering is investigating.

API failures in all regions (US, Europe, and India)

Jan <var data-var='date'> 3</var>, <var data-var='time'>23:19</var> EST Resolved - This incident has been resolved. Jan <var data-var='date'> 3</var>, <var data-var='time'>18:24</var> EST Monitoring - We have implemented a fix and are monitoring the system. Jan <var data-var='date'> 3</var>, <var data-var='time'>17:51</var> EST Identified - We are experiencing slowness in report generation. The investigation is ongoing, and we hope to have this resolved shortly.

Degraded performance for report generation.

Nov <var data-var='date'> 9</var>, <var data-var='time'>12:40</var> EST Resolved - UI failing to render for all customers

UI failing to load

Jan <var data-var='date'>13</var>, <var data-var='time'>12:43</var> EST Resolved - Summary On Thursday, January 13th, 2022 between 12:43 PM EST to 6:16 PM EST, several areas of the ChartHop platform was experiencing significant slowdowns due to a high volume of traffic in the database. Impact The impact was slowness and inability to use the platform its full functionality for approximately 5.5 hours. Resolution ChartHop’s DevOps teams increased the size of the server to process the backup traffic which restored the full functionality of the platform. Steps of prevention Install more indexes that will prevent future traffic issues on our servers. Additionally, introduce roadmap items on the data model to reduce reliance on specific servers to further improve performance.

Is ChartHop down?

ChartHop status is UP

Lastest published events

System maintenance

System maintenance

ChartHop is not loading

ChartHop Services, Components and Regions

Recent ChartHop events

System maintenance

System maintenance

ChartHop is not loading

ChartHop API is not responding

MongoDB incident impacting ChartHop

High error rates on UI