Feb <var data-var='date'>17</var>, <var data-var='time'>09:52</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>17</var>, <var data-var='time'>08:18</var> UTC Identified - We are working on the issue and will update next steps in few mins.

Verloop bots on India / US region are down.

Jan <var data-var='date'> 8</var>, <var data-var='time'>11:08</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'> 8</var>, <var data-var='time'>10:43</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jan <var data-var='date'> 8</var>, <var data-var='time'>10:34</var> UTC Update - We are continuing to work on a fix for this issue. Jan <var data-var='date'> 8</var>, <var data-var='time'>10:24</var> UTC Update - We are continuing to work on a fix for this issue. Jan <var data-var='date'> 8</var>, <var data-var='time'>10:23</var> UTC Identified - We are seeing network related issues on the MENA region of the Verloop. Team is working on it on priority.

MENA Region BOTS and Agent Portal is down.

Dec <var data-var='date'>12</var>, <var data-var='time'>04:57</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'>12</var>, <var data-var='time'>02:49</var> UTC Monitoring - The issue has been resolved and we are monitoring the recovery of GRAPH API's of whatsapp. Dec <var data-var='date'>11</var>, <var data-var='time'>18:51</var> UTC Investigating - We are seeing increased number of errors from META API for sending messages and getting updates from META. We are closely monitoring the situation and will update soon on the developments.

Global Whatsapp messaging issues.

Dec <var data-var='date'> 9</var>, <var data-var='time'>09:04</var> UTC Resolved - This issue has been resolved. Dec <var data-var='date'> 9</var>, <var data-var='time'>08:33</var> UTC Update - We are getting the services failover to back up instances. Dec <var data-var='date'> 9</var>, <var data-var='time'>07:54</var> UTC Identified - We are working on the issue with the DB on Verloop side, we are trying to restore services as of now.

Degraded performance on chat and voice calls.

Nov <var data-var='date'>27</var>, <var data-var='time'>15:26</var> UTC Resolved - The issue has been resolved. Nov <var data-var='date'>27</var>, <var data-var='time'>15:14</var> UTC Investigating - We are seeing errors on the whatsapp business management. This does not impact sending messages or existing outreaches. The impact is on fetching / creating new templates or flows. Team is investigating the issue and will update on progress.

Whatsapp business management api's failing (EditTemplate /  Fetchall flows / create template / fetch template etc)

Sep <var data-var='date'> 2</var>, <var data-var='time'>02:02</var> UTC Resolved - This incident has been resolved. Sep <var data-var='date'> 2</var>, <var data-var='time'>02:01</var> UTC Update - This was a partial outage for some customers. Sep <var data-var='date'> 2</var>, <var data-var='time'>02:00</var> UTC Update - We are continuing to investigate this issue. Sep <var data-var='date'> 2</var>, <var data-var='time'>02:00</var> UTC Update - We are continuing to investigate this issue. Sep <var data-var='date'> 2</var>, <var data-var='time'>00:30</var> UTC Investigating - All the Whatsapp messaging on Verloop GCP platform is affected due to issue on the META API. We are working with META team closely to resolve the issue.

Whatsapp messaging is affected.

Aug <var data-var='date'> 2</var>, <var data-var='time'>02:49</var> UTC Resolved - All the channels are back to normal. We will be sharing a detailed RCA in sometime. Aug <var data-var='date'> 2</var>, <var data-var='time'>00:29</var> UTC Monitoring - We have pushed a fix for the issue and bots are responding fine, we are currently monitoring. Aug <var data-var='date'> 1</var>, <var data-var='time'>09:53</var> UTC Update - This is only affecting customers in specific region, this would not impact Middle east customers. Aug <var data-var='date'> 1</var>, <var data-var='time'>09:52</var> UTC Identified - Verloop bots are not responding to customer chats. We are working on the issue on high priority. We know the root cause of the issue to be the queuing service.

BOTs are not responding to customer chat

Jul <var data-var='date'>31</var>, <var data-var='time'>10:52</var> UTC Resolved - The issues from Azure side are resolved, Verloop services are back up. We are constantly monitoring. Jul <var data-var='date'>31</var>, <var data-var='time'>05:36</var> UTC Update - We are closely working with microsoft team. Last update from Microsoft: Current Status: Automatic monitors detected a thermal anomaly. As a proactive measure, resources were powered down, causing a drop in availability. This manifested in connectivity failures for Virtual Machines for a subset of customers. We have successfully restored cooling systems, and temperatures have returned to normal operating thresholds. We have commenced a structured power- up sequence for previously powered- down compute and storage resources and are actively monitoring as we work through this extended process. Jul <var data-var='date'>30</var>, <var data-var='time'>19:53</var> UTC Update - We are still working with MS Azure team on getting our internal services back up and running. As the UAE region DB services are impacted the DR and Primary both DB are not reachable. We are working closely with Azure team to get the issue resolved. Jul <var data-var='date'>30</var>, <var data-var='time'>17:36</var> UTC Update - Azure network issue still seems to be having issues. We are still seeing issues with the internal services including in the DR zone are not reachable. We are trying other options to restart the Verloop chatbot services. We appreciate your patience. Jul <var data-var='date'>30</var>, <var data-var='time'>16:35</var> UTC Update - We are continuing to monitor for any further issues. Jul <var data-var='date'>30</var>, <var data-var='time'>14:26</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>30</var>, <var data-var='time'>14:26</var> UTC Investigating - Due to Global issues with the Microsoft Azure network services (https://azure.status.microsoft/en-gb/status), Verloop customers of Middle east region might face intermittent issues. We are closely monitoring the situation and will inform on further action soon.

Middle east customers might face some intermittent issues due to global network outage of MS Azure.

Jul <var data-var='date'>28</var>, <var data-var='time'>09:42</var> UTC Resolved - The issue has been identified and resolved. We will reporting the RCA on this in coming days. Jul <var data-var='date'>28</var>, <var data-var='time'>09:27</var> UTC Investigating - We are seeing bot responses being slow or non responsive in some cases. Team is working on resolving the issue.

Bot responses are slow or non responsive

May <var data-var='date'>18</var>, <var data-var='time'>03:50</var> UTC Resolved - This incident has been resolved. May <var data-var='date'>18</var>, <var data-var='time'>03:05</var> UTC Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'>18</var>, <var data-var='time'>03:05</var> UTC Investigating - Google DNS changes on the networking level has made some of the critical services of Verloop being inaccissible. Team is working to resolve the issue.

Issue with Whatsapp bots and Reporting

Apr <var data-var='date'>27</var>, <var data-var='time'>01:41</var> UTC Resolved - BOTS Are now responding to customers. Issue has been resolved. Detailed RCA to follow. Apr <var data-var='date'>27</var>, <var data-var='time'>01:32</var> UTC Investigating - We are seeing higher error rates in MENA region which is affecting BOT responses to customers. We are investigating the issue.

Experiencing delays and higher error rates for BOTS in MENA region.

Apr <var data-var='date'> 5</var>, <var data-var='time'>08:07</var> UTC Resolved - The DB load has come down and we are now seeing services are functioning normally. Apr <var data-var='date'> 5</var>, <var data-var='time'>06:56</var> UTC Investigating - We are currently investigating the issue.

Degraded performance on chat assignment and latencies in messages.

Nov <var data-var='date'>14</var>, <var data-var='time'>12:45</var> UTC Resolved - Issue has been resolved. We are working on updating the root cause here. Nov <var data-var='date'>14</var>, <var data-var='time'>12:39</var> UTC Identified - We have identified the issue, we are working on fix. Nov <var data-var='date'>14</var>, <var data-var='time'>12:10</var> UTC Investigating - We are investigating the issue. This issue is limited to UAE region customers only.

Chats on all channels are showing degraded performance for UAE customers.

Sep <var data-var='date'>26</var>, <var data-var='time'>11:57</var> UTC Resolved - The incident has been resolved. Sep <var data-var='date'>26</var>, <var data-var='time'>11:28</var> UTC Investigating - We are currently investigating the issue.

Realtime dashboard widgets taking longer time to load or failing to load for the Asia region customers.

Aug <var data-var='date'> 4</var>, <var data-var='time'>08:45</var> UTC Resolved - Update from WhatsApp The fix for the major disruption impacting sending messages on WhatsApp Cloud API has been deployed and this issue is now mitigated as of 3rd Aug 2023 11:50 PM PDT. Thank you for your patience. Aug <var data-var='date'> 4</var>, <var data-var='time'>06:36</var> UTC Monitoring - WhatsApp Cloud API is experiencing major disruption impacting sending messages. This issue started at 3rd Aug 2023 6:30 PM PDT. WhatsApp engineering teams have identified the root cause of the issue. Whatsapp engineering team is currently deploying the fix and the expected deployment time is in 6 hours. We will provide another update within 4 hours or sooner if additional information is available. This impacts all the regions for the WhatsApp message delivery.

Whatsapp Business messages has a issue.

Jul <var data-var='date'> 4</var>, <var data-var='time'>08:45</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 4</var>, <var data-var='time'>08:42</var> UTC Update - We are continuing to monitor for any further issues. Jul <var data-var='date'> 4</var>, <var data-var='time'>08:42</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'> 4</var>, <var data-var='time'>08:42</var> UTC Update - We are continuing to investigate this issue. Jul <var data-var='date'> 4</var>, <var data-var='time'>08:21</var> UTC Investigating - We are currently investigating the issue with the bots not responding to chats on GCP. This does not affect customers hosted on Azure.

Chat bots are not responding to the chats.

Jun <var data-var='date'>27</var>, <var data-var='time'>04:30</var> UTC Resolved - The WOXY service, which is responsible for connecting to WhatsApp, encountered an issue where it became unresponsive and unable to establish a connection. This problem arose due to a network glitch that occurred within our subnet, disrupting the normal functioning of the service. Consequently, the WOXY service entered a crashloop backoff state within our Kubernetes cluster. In order to resolve this issue, we took appropriate action and initiated a restart of the WOXY service. By doing so, we aimed to bring the service back online and restore its normal functionality. This restart was necessary to rectify the underlying problem caused by the network glitch and ensure that the WOXY service could once again establish a stable connection to WhatsApp.

Whatsapp bots not responding.

Apr <var data-var='date'> 7</var>, <var data-var='time'>11:45</var> UTC Resolved - Empathize with those affected and let them know everything is operating as normal. Apr <var data-var='date'> 7</var>, <var data-var='time'>11:35</var> UTC Monitoring - Let your users know once a fix is in place, and keep communication clear and precise. Apr <var data-var='date'> 7</var>, <var data-var='time'>11:41</var> UTC Identified - As you continue to work through the incident, update your customers frequently. Apr <var data-var='date'> 7</var>, <var data-var='time'>12:11</var> UTC Investigating - When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.

Is verloop down?

verloop status is UP

Lastest published events

Verloop bots on India / US region are down.

MENA Region BOTS and Agent Portal is down.

Global Whatsapp messaging issues.

verloop Services, Components and Regions

Recent verloop events

Verloop bots on India / US region are down.

MENA Region BOTS and Agent Portal is down.

Global Whatsapp messaging issues.

Degraded performance on chat and voice calls.

Whatsapp business management api's failing (EditTemplate / Fetchall flows / create template / fetch template etc)

Whatsapp messaging is affected.