Aug <var data-var='date'>17</var>, <var data-var='time'>14:55</var> CEST Resolved - This incident has been resolved. Aug <var data-var='date'>17</var>, <var data-var='time'>14:32</var> CEST Investigating - We are currently investigating this issue.

Plugins, Webhooks, and Handovers are failing

Jul <var data-var='date'>28</var>, <var data-var='time'>13:30</var> CEST Resolved - Around 13:30 CEST tere was a small bug introduced in a maintenance patch for our Genesys integration, that unfortunately affected all handovers. The bug was quickly solved after discovery around 14:30, and we have introduced regression test to avoid the issue in the future. We apologise for the inconvenience and we will do our best to make sure it doesn't happen again.

Genesys Handover / Livechat issue

Jul <var data-var='date'>22</var>, <var data-var='time'>10:30</var> CEST Resolved - Our Voicebot providers are facing performance issues causing certain calls to drop during conversations. We are doing our best to troubleshoot the situation and restore performance. Update: A fix was applied at 14:55, and we are monitoring the situation

Performance issues for Voicebot

Jun <var data-var='date'>11</var>, <var data-var='time'>20:01</var> CEST Resolved - Performance is stable and the incident is resolved. Jun <var data-var='date'>11</var>, <var data-var='time'>17:45</var> CEST Monitoring - A fix has been implemented, and we are seeing improved performance. We continue monitoring the situation. Jun <var data-var='date'>11</var>, <var data-var='time'>17:25</var> CEST Identified - Our providers are having performance issues causing degraded bot-performance. We are monitoring the situation and doing our best to restore performance

Performance-issues with AI-matching causing messaging delivery issues and increased fallback rates

Jun <var data-var='date'>11</var>, <var data-var='time'>14:37</var> CEST Completed - Database maintenance preparing for Markets is complete. All delayed analytics dashboards and events looks fully synced, and systems are operating normally. No data was lost. Thank you for your patience. Jun <var data-var='date'>11</var>, <var data-var='time'>10:11</var> CEST Update - - Background We are finalizing a database operation in preparation for Markets. - Current Impact Analytics dashboards and events are currently up to date as of June 11th at 5:00 AM. The remaining events will sync once the current operation concludes. No data has been lost, only delayed. - Current Progress We expect the majority of events to be fully updated by 12:00 PM, with the exception of message events which could take a bit longer due to the size of them. Jun <var data-var='date'>11</var>, <var data-var='time'>08:24</var> CEST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>11</var>, <var data-var='time'>08:23</var> CEST Scheduled - We are currently resyncing analytics after Maintenance, we will update as soon as all tables are fully resynced.

Analytics Maintenance - Resync of data ongoing

Jun <var data-var='date'> 2</var>, <var data-var='time'>10:03</var> CEST Completed - Scheduled maintenance is completed and all statistics tables are fully re-synced. Thank you for the patience. Jun <var data-var='date'> 1</var>, <var data-var='time'>13:26</var> CEST Verifying - Maintainace is done, and we are waiting for all tables to resync Jun <var data-var='date'> 1</var>, <var data-var='time'>12:42</var> CEST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'> 1</var>, <var data-var='time'>12:41</var> CEST Scheduled - The analytics page will experience downtime for approximately 30 minutes due to maintenance.

Analytics maintenance

May <var data-var='date'>27</var>, <var data-var='time'>17:51</var> CEST Resolved - Performance are restored and stable. May <var data-var='date'>27</var>, <var data-var='time'>17:00</var> CEST Monitoring - Function is restored and we are monitoring the situation. May <var data-var='date'>27</var>, <var data-var='time'>16:58</var> CEST Update - We are still working on a fix to restore function and working as hard as we can. May <var data-var='date'>27</var>, <var data-var='time'>16:32</var> CEST Identified - The issue is identified and we are working on a solution to restore performance.

May <var data-var='date'>20</var>, <var data-var='time'>22:47</var> CEST Resolved - This incident has been resolved. May <var data-var='date'>20</var>, <var data-var='time'>15:21</var> CEST Monitoring - We have implemented a fix and seeing improved performance. We continue monitoring the behavior May <var data-var='date'>20</var>, <var data-var='time'>15:08</var> CEST Update - We are still working on the issue causing failed message delivery and error codes in several bots. Coming back with an update as soon as possible May <var data-var='date'>20</var>, <var data-var='time'>14:42</var> CEST Update - We are still experiencing inconsistent performance issues with AI-matching and Bot-outpus. Working as hard as we can to fix May <var data-var='date'>20</var>, <var data-var='time'>13:40</var> CEST Update - We are still working on restoring optimal performance May <var data-var='date'>20</var>, <var data-var='time'>11:30</var> CEST Identified - We are working as hard as we can to restore performance

Performance-issues with AI-matching causing increased fallback rates

May <var data-var='date'>19</var>, <var data-var='time'>20:39</var> CEST Resolved - This incident has been resolved. May <var data-var='date'>19</var>, <var data-var='time'>18:07</var> CEST Update - The behaviour looks more stable, and we are monitoring the behaviour. May <var data-var='date'>19</var>, <var data-var='time'>18:07</var> CEST Update - We are continuing to monitor for any further issues. May <var data-var='date'>19</var>, <var data-var='time'>17:49</var> CEST Monitoring - The behaviour looks more stable, and we are monitoring the situation. May <var data-var='date'>19</var>, <var data-var='time'>16:13</var> CEST Identified - The behaviour seems inconsistent, while we are working on a fix May <var data-var='date'>19</var>, <var data-var='time'>14:18</var> CEST Investigating - We are experiencing LLM-issues that could affect AI-generated answers and fallback rates

LLM issues affecting AI-generated answers

Mar <var data-var='date'>12</var>, <var data-var='time'>13:05</var> CET Resolved - The incident have been resolved. Platform is back up and stable. There will be an internal post-mortem incident report. Mar <var data-var='date'>12</var>, <var data-var='time'>12:43</var> CET Monitoring - A fix has been implemented and we are monitoring the results Mar <var data-var='date'>12</var>, <var data-var='time'>12:35</var> CET Update - We are continuing investigating this issue, follow this page for updates Mar <var data-var='date'>12</var>, <var data-var='time'>12:13</var> CET Update - We are continuing to investigate this issue. Mar <var data-var='date'>12</var>, <var data-var='time'>12:09</var> CET Investigating - Platform is currently down, we are investigating

Platform is down

Mar <var data-var='date'> 8</var>, <var data-var='time'>21:10</var> CET Resolved - This incident has been resolved. Mar <var data-var='date'> 8</var>, <var data-var='time'>13:02</var> CET Monitoring - A fix has been implemented and we are monitoring the results Mar <var data-var='date'> 7</var>, <var data-var='time'>10:00</var> CET Identified - The issue has been identified and we are implementing a fix

Partial downtime in analytics

Feb <var data-var='date'>11</var>, <var data-var='time'>11:33</var> CET Resolved - Incident resolved Feb <var data-var='date'>11</var>, <var data-var='time'>11:31</var> CET Investigating - We are currently investigating this issue.

Analytics in Bot Dashboard is down

Jan <var data-var='date'>27</var>, <var data-var='time'>16:07</var> CET Resolved - This incident has been resolved. Jan <var data-var='date'>27</var>, <var data-var='time'>12:36</var> CET Monitoring - A fix has been implemented and we are getting reports of working bots, we are still monitoring Jan <var data-var='date'>27</var>, <var data-var='time'>12:26</var> CET Identified - Because of some reliability changes we are experiencing problems between bubble and server connection, We identified the reason and we are trying to get it out as soon as possible. Due to how browser caching works, it will take some time to get it effective. Thanks for your patience.

Communication between chat clients and our servers is overloaded because of a reliability update

Jan <var data-var='date'>27</var>, <var data-var='time'>12:23</var> CET Resolved - This incident has been resolved. Jan <var data-var='date'>26</var>, <var data-var='time'>21:31</var> CET Monitoring - We are doing a migration on our databases for the Kindly Inbox website. If you are having connection troubles like messages not appearing unless a refresh, we are suggesting logging off and logging in again. If the problems continue please notify us.

Migration on Kindly Inbox Databases

Jan <var data-var='date'> 7</var>, <var data-var='time'>09:37</var> CET Resolved - The issue was resolved by the vendor at 21:40 6th of January Jan <var data-var='date'> 6</var>, <var data-var='time'>18:34</var> CET Monitoring - Partially solved by the vendor. Several tests show normal behaviour, but it can still occur that the dashboards fail to initialise. Jan <var data-var='date'> 6</var>, <var data-var='time'>16:05</var> CET Identified - One of our providers responsible for loading our dashboards are having issues atm causing dashboards in your bot not to load. They are aware and working on a fix. No data will be lost. It is only affecting visually loading the data.

Analytics dashboards fails to load

Nov <var data-var='date'>18</var>, <var data-var='time'>16:03</var> CET Resolved - From Cloudflare: A fix has been implemented and we believe the incident is now resolved. We are continuing to monitor for errors to ensure all services are back to normal. Nov <var data-var='date'>18</var>, <var data-var='time'>13:36</var> CET Update - Update from Cloudflare: We are seeing services recover, but customers may continue to observe higher-than-normal error rates as we continue remediation efforts. Nov <var data-var='date'>18</var>, <var data-var='time'>13:29</var> CET Update - Working again. Calling it "partial outage" until Cloudflare is fully up and running. Nov <var data-var='date'>18</var>, <var data-var='time'>13:27</var> CET Identified - Down again. Same reason. Nov <var data-var='date'>18</var>, <var data-var='time'>13:12</var> CET Monitoring - All systems seems to be back up, but Claudflare still reports error. Assuming our dependency is solved, while others still waiting. Will monitor going forward. Nov <var data-var='date'>18</var>, <var data-var='time'>13:05</var> CET Update - Bots respond if you managed to load the widget before cloudflare stopped working. From cloudflare: Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available. Nov <var data-var='date'>18</var>, <var data-var='time'>13:01</var> CET Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>18</var>, <var data-var='time'>13:01</var> CET Identified - Cloudflare has issues, causing part of Kindly among many other services to be down: https://www.cloudflarestatus.com/

Platform & Chat widget is not accessible

Oct <var data-var='date'>28</var>, <var data-var='time'>12:16</var> CET Resolved - All systems stable. Oct <var data-var='date'>28</var>, <var data-var='time'>11:20</var> CET Monitoring - All systems seems to be back up now. Monitoring to make sure. Oct <var data-var='date'>28</var>, <var data-var='time'>11:06</var> CET Update - Deploying a potential fix and purging the cache. Oct <var data-var='date'>28</var>, <var data-var='time'>10:43</var> CET Update - Working on the workaround. Oct <var data-var='date'>28</var>, <var data-var='time'>10:20</var> CET Update - Working on the workaround. Oct <var data-var='date'>28</var>, <var data-var='time'>10:05</var> CET Update - Working on the workaround. Oct <var data-var='date'>28</var>, <var data-var='time'>09:43</var> CET Update - We have confirmed that Telenor is aware of the issue and have internal resources on it. We continue on our workaround in parallel. Oct <var data-var='date'>28</var>, <var data-var='time'>09:40</var> CET Update - Looks to be more companies than Kindly that are affected. Still working on the workaround. Oct <var data-var='date'>28</var>, <var data-var='time'>09:31</var> CET Update - Working on workarounds while getting in contact with Telenor. Oct <var data-var='date'>28</var>, <var data-var='time'>09:13</var> CET Identified - We have identified that when using Telenor as a network provider, our bots will not respond, nor access our platform. We are working on finding out why.

Bots & Platform not responding when using Telenor networks

Aug <var data-var='date'>27</var>, <var data-var='time'>16:47</var> CEST Resolved - An employee’s laptop was inadvertently connected directly to their home internet service, bypassing a router/firewall and exposing network ports to the public internet. Automated bots scanning the internet identified an exposed service running inside a Docker container and exploited a vulnerability to deploy a cryptominer within that container. The container held some chatbot configuration files only; it did not contain sensitive or personal data. Our investigation to date shows no evidence of data leakage or any other breach beyond the cryptominer. We are reinforcing endpoint and network configuration requirements to prevent recurrence.

Isolated Workstation Configuration Issue

Apr <var data-var='date'>14</var>, <var data-var='time'>12:00</var> CEST Resolved - This incident has been resolved. Apr <var data-var='date'>14</var>, <var data-var='time'>11:51</var> CEST Monitoring - Issue seemed to be solved. Currently monitoring and understanding what caused the issue. Apr <var data-var='date'>14</var>, <var data-var='time'>11:32</var> CEST Update - We are continuing to investigate this issue. Apr <var data-var='date'>14</var>, <var data-var='time'>11:29</var> CEST Investigating - We are currently investigating the issue.

Handover to human chat are failing

Apr <var data-var='date'>11</var>, <var data-var='time'>16:48</var> CEST Completed - The scheduled maintenance has been completed. Apr <var data-var='date'>11</var>, <var data-var='time'>16:35</var> CEST Verifying - Verification is currently underway for the maintenance items. Apr <var data-var='date'>11</var>, <var data-var='time'>16:00</var> CEST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'> 8</var>, <var data-var='time'>09:13</var> CEST Scheduled - There will be some upgrades to our infrastructure that might cause intermittent connection issues

Infrastructure Updates

Mar <var data-var='date'>18</var>, <var data-var='time'>15:02</var> CET Resolved - This incident has been resolved. Mar <var data-var='date'>18</var>, <var data-var='time'>14:43</var> CET Update - No clear reason yet, still investigating Mar <var data-var='date'>18</var>, <var data-var='time'>11:45</var> CET Investigating - Currently investigating the issue.

Kindly Inbox (Live chat) responds slowly

Mar <var data-var='date'>13</var>, <var data-var='time'>15:00</var> CET Resolved - This incident has been resolved. Handover to humans had been failing for 14 minutes.

Jan <var data-var='date'>16</var>, <var data-var='time'>19:47</var> CET Resolved - After a normal business day we have looked at the numbers and they seem to be normal. We will continue to monitor the sessions numbers to correct any discrepancies that might have occurred because of the database update issue. Jan <var data-var='date'>16</var>, <var data-var='time'>01:15</var> CET Monitoring - It has been determined that there is a bug in the analytics database that affects the behavior for how Kindly calculates sessions. We have an open ticket with the database provider where they have acknowledged this bug and are working on a fix, but they don't know how long it will take. In order to restore most of the functionality of the database we have disabled a part of the sessions calculation before resuming normal analytics operations. This will affect stats in dashboards that measure sessions. We will resume ingesting analytics events to still have accurate statistics about messages, fallbacks, labels and all stats that are not derived from sessions. Once a bugfix is available, we plan use the accurate data to recreate the sessions data for the affected period and have accurate sessions statistics, but in the meantime the charts that show sessions may not reflect accurate numbers. Jan <var data-var='date'>15</var>, <var data-var='time'>18:20</var> CET Identified - During a routing maintenance upgrade of the analytics and unexpected event happened that requires us to reboot the database service. While this is ongoing analytics dashboards may be unavailable for some time. During this, all chatbot operations work as normal and all analytics events are recorded in a queue and no data is lost. Only the ability to view the data will be temporarily unavailable.

Upgrading analytics database

Jan <var data-var='date'>13</var>, <var data-var='time'>10:00</var> CET Resolved - Minor degredation to the analytics pages of Kindly due to partial outage in Kindly's analytics vendor.

Minor degredation in analytics pages

Dec <var data-var='date'>11</var>, <var data-var='time'>16:10</var> CET Resolved - Database is online, dashboards and APIs function as normal, all queued analytics events have been written to the database. Dec <var data-var='date'>11</var>, <var data-var='time'>15:36</var> CET Monitoring - The database is now back online and it is possible to make queries again. We will now start writing the queued events to the database. Dec <var data-var='date'>11</var>, <var data-var='time'>15:25</var> CET Update - We are continuing to work on a fix for this issue. Dec <var data-var='date'>11</var>, <var data-var='time'>15:21</var> CET Identified - While testing out some new features of the analytics systems we accidentally triggered some downtime for the analytics database. Chats are unaffected, and the platform app is working except you can't currently view analytics. APIs for analytics are also temporarily unavailable. No data is lost, all analytical data is being recorded and waiting in a queue to be written to the analytics database once it is online again.

Is Kindly down?

Kindly status is UP

Lastest published events

Plugins, Webhooks, and Handovers are failing

Genesys Handover / Livechat issue

Performance issues for Voicebot

Kindly Services, Components and Regions

Recent Kindly events

Plugins, Webhooks, and Handovers are failing

Genesys Handover / Livechat issue

Performance issues for Voicebot

Performance-issues with AI-matching causing messaging delivery issues and increased fallback rates

Analytics Maintenance - Resync of data ongoing

Analytics maintenance