Jul <var data-var='date'>22</var>, <var data-var='time'>09:05</var> CDT Resolved - This incident has been resolved. Jul <var data-var='date'>22</var>, <var data-var='time'>08:51</var> CDT Monitoring - A fix has been implemented, and we are monitoring the results. Jul <var data-var='date'>22</var>, <var data-var='time'>08:14</var> CDT Investigating - We are currently investigating this issue.

US Webhook Queue Delays

Jun <var data-var='date'> 1</var>, <var data-var='time'>18:07</var> CDT Resolved - This incident has been resolved. Jun <var data-var='date'> 1</var>, <var data-var='time'>15:04</var> CDT Monitoring - We are currently monitoring Jun <var data-var='date'> 1</var>, <var data-var='time'>14:04</var> CDT Update - Queues are still processing after implementing the fix. Jun <var data-var='date'> 1</var>, <var data-var='time'>12:00</var> CDT Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'> 1</var>, <var data-var='time'>10:56</var> CDT Investigating - We are currently investigating this issue.

US - Job Processing Delayed

May <var data-var='date'> 1</var>, <var data-var='time'>14:42</var> CDT Resolved - This incident has been resolved. May <var data-var='date'> 1</var>, <var data-var='time'>13:31</var> CDT Monitoring - A fix has been implemented, and we are currently monitoring the results. May <var data-var='date'> 1</var>, <var data-var='time'>11:22</var> CDT Update - We are still actively troubleshooting this issue. May <var data-var='date'> 1</var>, <var data-var='time'>10:14</var> CDT Investigating - We are currently investigating this issue.

Apr <var data-var='date'>24</var>, <var data-var='time'>04:00</var> CDT Completed - The scheduled maintenance has been completed. Apr <var data-var='date'>23</var>, <var data-var='time'>23:00</var> CDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'>21</var>, <var data-var='time'>09:16</var> CDT Scheduled - This maintenance involves a backend network switch for job processing. We expect minimal customer impact during the maintenance window, and the web application will remain accessible. Job processing may be delayed during this time. New IP address ranges added: 20.15.246.160/28 and 20.84.143.0/28. For more details: https://docs.sonar.expert/networking/sonar-ip-addressing

US - Infrastructure Maintenance

Apr <var data-var='date'>10</var>, <var data-var='time'>14:25</var> CDT Resolved - This incident has been resolved. Apr <var data-var='date'>10</var>, <var data-var='time'>13:08</var> CDT Monitoring - A fix has been implemented, and we are monitoring the results. Apr <var data-var='date'>10</var>, <var data-var='time'>11:19</var> CDT Investigating - We are currently investigating this issue with jobs being delayed.

Mar <var data-var='date'> 2</var>, <var data-var='time'>01:44</var> CST Resolved - This incident has been resolved. Mar <var data-var='date'> 1</var>, <var data-var='time'>17:24</var> CST Monitoring - Currently monitoring. The application is currently stable at this time. Mar <var data-var='date'> 1</var>, <var data-var='time'>14:48</var> CST Identified - Currently the application is stable. We are continuing the monitoring and actively working on a permanent solution for this issue. Mar <var data-var='date'> 1</var>, <var data-var='time'>11:25</var> CST Investigating - We are currently investigating this issue. Mar <var data-var='date'> 1</var>, <var data-var='time'>10:14</var> CST Update - Additional measures have been taken to ensure system stability. Mar <var data-var='date'> 1</var>, <var data-var='time'>09:47</var> CST Update - We've discovered further issues and are taking action to resolve them. Mar <var data-var='date'> 1</var>, <var data-var='time'>09:12</var> CST Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'> 1</var>, <var data-var='time'>09:07</var> CST Identified - The issue has been identified and a fix is being implemented. Mar <var data-var='date'> 1</var>, <var data-var='time'>08:58</var> CST Investigating - We're aware of increased latency and unresponsive requests and are investigating the issue.

US Sonar Application Slow or Unresponsive

Mar <var data-var='date'> 1</var>, <var data-var='time'>04:47</var> CST Resolved - Application health is now back to normal. Mar <var data-var='date'> 1</var>, <var data-var='time'>03:32</var> CST Update - The root cause has been identified and a fix applied. The system is entering full recovery. Mar <var data-var='date'> 1</var>, <var data-var='time'>02:09</var> CST Update - We are monitoring partial recovery of the application in the US datacenter. Investigation into the root cause is ongoing. Mar <var data-var='date'> 1</var>, <var data-var='time'>01:22</var> CST Update - We are still investigating the root cause of application slowness and increased error rate affecting US data center. Mar <var data-var='date'> 1</var>, <var data-var='time'>00:00</var> CST Investigating - We are currently investigating this issue.

US Sonar Increase in Nginx Errors

Feb <var data-var='date'>26</var>, <var data-var='time'>14:08</var> CST Resolved - This incident has been resolved. Feb <var data-var='date'>26</var>, <var data-var='time'>13:04</var> CST Monitoring - The issue has been identified, and a fix has been implemented. Currently monitoring the results. Feb <var data-var='date'>26</var>, <var data-var='time'>12:23</var> CST Update - We are continuing to investigate this issue. Feb <var data-var='date'>26</var>, <var data-var='time'>10:46</var> CST Update - We have made changes and saw some improvements with latency. We are still investigating this issue. Feb <var data-var='date'>26</var>, <var data-var='time'>09:32</var> CST Investigating - We are currently investigating this issue.

US Sonar Slowness

Feb <var data-var='date'>25</var>, <var data-var='time'>13:34</var> CST Resolved - This incident has been resolved. Feb <var data-var='date'>25</var>, <var data-var='time'>12:09</var> CST Monitoring - A fix has been implemented, and we are monitoring the results. Feb <var data-var='date'>25</var>, <var data-var='time'>11:53</var> CST Identified - The issue has been identified, and a fix is being implemented. We will update once the fix deploy has completed. Feb <var data-var='date'>25</var>, <var data-var='time'>11:00</var> CST Investigating - We are currently investigating this issue.

Feb <var data-var='date'>19</var>, <var data-var='time'>20:15</var> CST Resolved - This incident has been resolved. Feb <var data-var='date'>19</var>, <var data-var='time'>10:53</var> CST Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>19</var>, <var data-var='time'>10:00</var> CST Investigating - We are currently experiencing an issue with the Auth0 SSO/SAML connection on the CA DC. Development has identified the issue and is working on a solution at this time.

CA - SSO/SAML Connection Issue

Feb <var data-var='date'>12</var>, <var data-var='time'>02:09</var> CST Completed - The scheduled maintenance has been completed. Feb <var data-var='date'>11</var>, <var data-var='time'>23:00</var> CST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>29</var>, <var data-var='time'>09:34</var> CST Scheduled - The application will be unavailable for approximately 30 to 90 minutes within the US DC during the maintenance window. Additionally, asynchronous workloads such as backend job processing will experience delays. Our team will work to minimize service impact and will restore full functionality as quickly as possible. Updates will be shared if there are any deviations from the expected timeline.

Feb <var data-var='date'> 3</var>, <var data-var='time'>19:11</var> CST Resolved - This incident has been resolved. Feb <var data-var='date'> 3</var>, <var data-var='time'>17:44</var> CST Investigating - We are currently investigating this issue. We are seeing an increase in nginx errors at this time.

Increase Nginx Errors

Jan <var data-var='date'>13</var>, <var data-var='time'>00:49</var> CST Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>12</var>, <var data-var='time'>23:00</var> CST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'> 9</var>, <var data-var='time'>13:11</var> CST Scheduled - The application will be unavailable for approximately 30 minutes within the CA DC during the maintenance window. Additionally, asynchronous workloads such as backend job processing will experience delays. Our team will work to minimize service impact and will restore full functionality as quickly as possible. Updates will be shared if there are any deviations from the expected timeline.

CA - Infrastructure Maintenance

Dec <var data-var='date'>30</var>, <var data-var='time'>18:04</var> CST Resolved - This incident has been resolved. Dec <var data-var='date'>30</var>, <var data-var='time'>17:21</var> CST Monitoring - As of 3:30pm CST, we stopped seeing IPPay failures. We are currently monitoring at this time. Dec <var data-var='date'>30</var>, <var data-var='time'>16:32</var> CST Update - IPPay is still investigating at this time. If you experience any failed payments, if you attempt another payment, this will most likely be successful. The current failure rate is 20 to 30% for IPPay. No other payment processors are impacted. Dec <var data-var='date'>30</var>, <var data-var='time'>14:42</var> CST Investigating - We are seeing an increase in IPPay connection failures. We have reached out to IPPay, as we see 20% of the connections fail with IPPay.

IPPay Intermittent Connection Failures

Dec <var data-var='date'>16</var>, <var data-var='time'>13:58</var> CST Resolved - This incident has been resolved. Dec <var data-var='date'>16</var>, <var data-var='time'>13:24</var> CST Monitoring - A fix has been implemented, and we are monitoring the results. Dec <var data-var='date'>16</var>, <var data-var='time'>12:19</var> CST Identified - Reporting is currently delayed in updates due to a vendor issue. We have identified the issue and are working with the vendor to resolve this issue.

Reporting Delays

Dec <var data-var='date'>14</var>, <var data-var='time'>02:00</var> CST Completed - The scheduled maintenance has been completed. Dec <var data-var='date'>13</var>, <var data-var='time'>23:00</var> CST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>11</var>, <var data-var='time'>12:08</var> CST Scheduled - During this maintenance window, customers may experience brief interruptions. Background jobs and automated processes may run slower than usual, as these operations will be temporarily paused while we complete the maintenance.

US - Infrastructure Emergency Maintenance

Dec <var data-var='date'>11</var>, <var data-var='time'>07:59</var> CST Resolved - This incident has been resolved. Dec <var data-var='date'>11</var>, <var data-var='time'>04:54</var> CST Monitoring - A fix has been implemented, and we are monitoring the results. Dec <var data-var='date'>11</var>, <var data-var='time'>03:54</var> CST Identified - The issue has been identified, and a fix is being implemented. Dec <var data-var='date'>11</var>, <var data-var='time'>00:17</var> CST Investigating - We are currently investigating this issue.

Application Unavailable

Dec <var data-var='date'> 1</var>, <var data-var='time'>13:53</var> CST Resolved - This incident has been resolved. Dec <var data-var='date'> 1</var>, <var data-var='time'>12:38</var> CST Monitoring - A fix has been implemented, and we are monitoring the results. Dec <var data-var='date'> 1</var>, <var data-var='time'>12:14</var> CST Update - We have implemented a fix and are currently monitoring. The application is currently stable at this time. Dec <var data-var='date'> 1</var>, <var data-var='time'>10:35</var> CST Update - We are experiencing a high error rate for inbound web requests. We are working on mitigating this issue. Dec <var data-var='date'> 1</var>, <var data-var='time'>09:44</var> CST Update - The application is currently stable. We are working to bring the application back to normal operating capacity. Dec <var data-var='date'> 1</var>, <var data-var='time'>08:09</var> CST Update - We are continuing to work on a fix for this issue. Dec <var data-var='date'> 1</var>, <var data-var='time'>06:59</var> CST Identified - The issue has been identified, and a fix is being implemented. Dec <var data-var='date'> 1</var>, <var data-var='time'>05:42</var> CST Update - We are continuing to investigate this issue. Dec <var data-var='date'> 1</var>, <var data-var='time'>04:50</var> CST Update - We are continuing to investigate this issue. Dec <var data-var='date'> 1</var>, <var data-var='time'>04:15</var> CST Update - We are continuing to investigate this issue. Dec <var data-var='date'> 1</var>, <var data-var='time'>03:51</var> CST Investigating - We are currently investigating this issue.

Sonar Application Unavailable

Nov <var data-var='date'>27</var>, <var data-var='time'>03:03</var> CST Completed - The scheduled maintenance has been completed. Nov <var data-var='date'>26</var>, <var data-var='time'>23:00</var> CST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>18</var>, <var data-var='time'>10:20</var> CST Scheduled - We will be performing infrastructure maintenance in the CA region. During this window, the application will experience multiple periods of unavailability, with each interruption expected to last between 30 and 60 minutes. Additionally, asynchronous workloads such as backend job processing will experience delays. Our team will work to minimize service impact and will restore full functionality as quickly as possible. Updates will be shared if there are any deviations from the expected timeline.

Nov <var data-var='date'>27</var>, <var data-var='time'>00:12</var> CST Completed - The scheduled maintenance has been completed. Nov <var data-var='date'>26</var>, <var data-var='time'>19:00</var> CST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>25</var>, <var data-var='time'>16:06</var> CST Scheduled - During this maintenance window, customers may experience brief interruptions. Background jobs and automated processes may run slower than usual, as these operations will be temporarily paused while we complete the maintenance.

Nov <var data-var='date'>18</var>, <var data-var='time'>09:09</var> CST Resolved - This incident has been resolved. Nov <var data-var='date'>18</var>, <var data-var='time'>09:03</var> CST Monitoring - Cloudflare has implemented a fix and we are seeing improvement. We will continue to monitor. Nov <var data-var='date'>18</var>, <var data-var='time'>08:55</var> CST Update - We are continuing to monitor. Nov <var data-var='date'>18</var>, <var data-var='time'>08:54</var> CST Identified - We are currently experiencing degraded performance with sonarPay due to ongoing issues with Cloudflare. As a result, there may be difficulties when processing payments or adding/updating payment methods. We’re actively monitoring the situation and will provide updates as they become available.

sonarPay Degraded Performance

Oct <var data-var='date'>24</var>, <var data-var='time'>13:44</var> CDT Resolved - This incident has been resolved. Oct <var data-var='date'>24</var>, <var data-var='time'>10:10</var> CDT Monitoring - The application is stable and currently monitoring. Oct <var data-var='date'>24</var>, <var data-var='time'>10:01</var> CDT Investigating - We are currently investigating the issue.

Oct <var data-var='date'> 7</var>, <var data-var='time'>03:00</var> CDT Completed - The scheduled maintenance has been completed. Oct <var data-var='date'> 6</var>, <var data-var='time'>23:00</var> CDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>25</var>, <var data-var='time'>10:42</var> CDT Scheduled - We will be performing infrastructure maintenance in the US region. During this window, the application will experience multiple periods of unavailability, with each interruption expected to last between 30 and 60 minutes. Additionally, asynchronous workloads such as backend job processing will experience delays. Our team will work to minimize service impact and will restore full functionality as quickly as possible. Updates will be shared if there are any deviations from the expected timeline.

Oct <var data-var='date'> 3</var>, <var data-var='time'>12:33</var> CDT Resolved - This incident has been resolved. Oct <var data-var='date'> 3</var>, <var data-var='time'>12:00</var> CDT Update - The webhook queue is continuing to recover. We are continuing to monitor. Oct <var data-var='date'> 3</var>, <var data-var='time'>11:37</var> CDT Update - The webhook queue is beginning to recover. We are continuing to monitor. Oct <var data-var='date'> 3</var>, <var data-var='time'>11:15</var> CDT Monitoring - A fix has been implemented and we are monitoring the results. Oct <var data-var='date'> 3</var>, <var data-var='time'>11:04</var> CDT Identified - The issue has been identified and a fix is being implemented. Oct <var data-var='date'> 3</var>, <var data-var='time'>10:51</var> CDT Investigating - We are currently experiencing a delay in webhook processing and are actively investigating.

Webhook Processing Delay

Oct <var data-var='date'> 3</var>, <var data-var='time'>03:00</var> CDT Completed - The scheduled maintenance has been completed. Oct <var data-var='date'> 2</var>, <var data-var='time'>23:00</var> CDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>25</var>, <var data-var='time'>10:40</var> CDT Scheduled - We will be performing infrastructure maintenance in the CA region. During this window, the application will experience multiple periods of unavailability, with each interruption expected to last between 30 and 60 minutes. Additionally, asynchronous workloads such as backend job processing will experience delays. Our team will work to minimize service impact and will restore full functionality as quickly as possible. Updates will be shared if there are any deviations from the expected timeline.

Is Sonar down?

Sonar status is UP

Lastest published events

US Webhook Queue Delays

US - Job Processing Delayed

US - Job Processing Delayed

Sonar Services, Components and Regions

Recent Sonar events

US Webhook Queue Delays

US - Job Processing Delayed

US - Job Processing Delayed

US - Infrastructure Maintenance

US - Job Processing Delayed

US Sonar Application Slow or Unresponsive