Mar <var data-var='date'>10</var>, <var data-var='time'>11:46</var> EDT Resolved - This incident has been resolved. Klaviyo events are processing as expected now. Our backlog of events that were queued during the incident were processed and notifications were triggered as our events caught up. Mar <var data-var='date'> 9</var>, <var data-var='time'>15:04</var> EDT Monitoring - A fix has been deployed and we are monitoring results. Mar <var data-var='date'> 9</var>, <var data-var='time'>14:10</var> EDT Investigating - We are currently investigating an issue affecting event delivery to Klaviyo. Some customers may experience delays or failures with events being sent to Klaviyo. Our team has identified authentication-related errors beginning around March 7 and is actively working to determine the root cause and restore normal event delivery. We will provide updates as we learn more.

Klaviyo event delivery failures

Mar <var data-var='date'> 4</var>, <var data-var='time'>13:41</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>28</var>, <var data-var='time'>20:45</var> EST Monitoring - On February 26th, our primary database failed after its internal logging system ran out of allocated disk space. As a result, our recovery infrastructure automatically switched to a backup database to maintain uptime. However, we recently discovered that because the logging system failed, the backup database was approximately two days behind in replication (replication is normally near-instant). As a result, orders and shipment records processed during that window were not present after failover. We chose not to re-import those orders because doing so may trigger unwanted duplicate events. Tracking updates will fail for those orders as a result. TO RESOLVE THIS ISSUE, you can take the following action: If you're using one of our integrated apps to trigger notifications (Klaviyo, Attentive): 1) Place a flow filter on any flows that use the Malomo:ShopifyOrderCreated event (typically Order Confirmation emails) that checks to see if a Shopify order has been placed in the last day. If yes, then proceed to trigger the flow as normal. 2) Place a flow filter on any flows that use the Malomo:ShipmentCreated event (typically Just Shipped emails) that checks to see if a Shopify order has been fulfilled int he last day. If yes, then proceed to trigger the flow as normal. 3) Once steps 1 and 2 are complete, place an order tag on any order that was placed or fulfilled between Feb 24th and Feb 26th. The tag can be anything. This will cause us to sync the order and begin tracking shipments and triggering notifications as expected. If you're using Shopify to trigger your notifications: Place an order tag on any order the was placed or fulfilled between Feb 24th and Feb 26th. The tag can be anything. This will cause us to sync the order and begin tracking any shipments as expected.

Tracking update issues

Oct <var data-var='date'>17</var>, <var data-var='time'>20:00</var> EDT Resolved - All events should be processing as normal. Oct <var data-var='date'>17</var>, <var data-var='time'>14:33</var> EDT Monitoring - We have identified an issue that has caused a slow down in processing events. While all events should process successfully, some customers may experience delays in some orders and fulfillments displaying in Malomo and events reaching integrations. Our team is working to clear the queue and eliminate these delays quickly. We will share further updates as we continue to resolve the issue.

Delayed events

Aug <var data-var='date'>21</var>, <var data-var='time'>09:24</var> EDT Resolved - Our team has confirmed that the backup in event processing for the Klaviyo integration has been fully resolved. Events are now processing as expected. Aug <var data-var='date'>20</var>, <var data-var='time'>13:54</var> EDT Monitoring - Our team identified an issue with the Klaviyo integration connection that has caused a backup in event processing. Some accounts may see delayed events during this time. Our team has deployed a fix and is monitoring the reduction in the event backlog.

Klaviyo Integration Delayed Event Processing

Jun <var data-var='date'> 3</var>, <var data-var='time'>14:30</var> EDT Resolved - This issue has been resolved. Jun <var data-var='date'> 3</var>, <var data-var='time'>11:07</var> EDT Investigating - Our team has identified an issue with duplicate Shopify Order Created events being generated which may be sending duplicate events for Order created to connected integrations. We are currently investigating.

Duplicate Order Created Events

Apr <var data-var='date'>28</var>, <var data-var='time'>15:14</var> EDT Resolved - This incident has been resolved. Apr <var data-var='date'>28</var>, <var data-var='time'>13:28</var> EDT Update - We are continuing to monitor for any further issues. Apr <var data-var='date'>28</var>, <var data-var='time'>13:28</var> EDT Monitoring - This issue has been fixed and our dashboard is now loading the orders list. Our team is continuing to monitor while our system indexes a backlog of orders to display. We expect to fully resolve this issue shortly. Apr <var data-var='date'>28</var>, <var data-var='time'>11:18</var> EDT Identified - Our team has identified an issue with slow load times in the dashboard Orders list, which is preventing orders and order history from being shown. Our team is currently working to fix this issue. No order information has been lost and all events are continuing to process as expected; the issue is only preventing the orders list from loading properly.

Dashboard Orders List

Mar <var data-var='date'>11</var>, <var data-var='time'>14:00</var> EDT Resolved - Our team was alerted to the an issue with our Klaviyo integration and upon investigation, we identified the integration failure and deployed a fix, restoring normal event processing. Events that were delayed during this period of time have now processed

Feb <var data-var='date'>18</var>, <var data-var='time'>14:10</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>18</var>, <var data-var='time'>11:09</var> EST Monitoring - Our team has fixed the issue and our platform is now processing shipping update events that were queued up from the start of the issue. These events are being sent to integrations as expected and shipping notifications will continue to send. Our team will continue to monitor shipping update events throughout the day. Feb <var data-var='date'>18</var>, <var data-var='time'>11:04</var> EST Identified - Our team has identified an issue with our third party carrier aggregator, which prevented shipping updates for certain carriers to be sent to Malomo. This may have impacted the delivery of shipping update notifications, beginning on February 16 to now, due to decreased events sent to our integrations. Our team is currently deploying a fix.

Delayed Carrier Shipment Updates

Feb <var data-var='date'> 3</var>, <var data-var='time'>06:00</var> EST Resolved - Our OAuth2 authentication service was returning unauthorized (401) status codes, causing API calls that require authentication to fail. This issue affected our ability to receive new orders and fulfillment updates from Shopify. The root cause was identified upon investigation as an issue within the OAuth2 service. A restart of the service successfully restored functionality. To enhance our response to similar issues in the future, we have implemented proactive alerting with our service provider to ensure faster detection and resolution of authentication failures. We are also backfilling these orders during the incident. We apologize for any inconvenience this may have caused and appreciate your patience as we continue to improve our system reliability.

OAuth2 Authentication Service Outage

Nov <var data-var='date'>30</var>, <var data-var='time'>13:06</var> EST Resolved - After close monitoring, we're continuing to see our events processing quickly to integrations without delays. Nov <var data-var='date'>29</var>, <var data-var='time'>20:12</var> EST Monitoring - We are seeing the queue processing events with minimal delay, and continuing to monitor it. Nov <var data-var='date'>29</var>, <var data-var='time'>14:35</var> EST Identified - We are currently observing minor delays in event processing. While all events are being processed successfully, some customers may experience slight delays in their events reaching integrations. Our team is working to clear the queue and eliminate these delays quickly. We will share further updates as we continue to resolve the issue.

Intermittent Shopify Order Created Event Delays

Aug <var data-var='date'> 8</var>, <var data-var='time'>12:32</var> EDT Resolved - The connectivity issues have been resolved. Aug <var data-var='date'> 1</var>, <var data-var='time'>11:33</var> EDT Identified - Our team has identified intermittent connectivity issues via the Shopify API. This is causing our Tracking Page Creator to be slow to load, as our system waits for theme information from Shopify. We're working with Shopify Partner Support to address the issue. Jul <var data-var='date'>31</var>, <var data-var='time'>15:27</var> EDT Investigating - We are currently investigating a connectivity issue with our Tracking Page Creator feature. The Tracking Page Creator in our dashboard may be slow to load or may fail to show existing tracking pages created in Shopify themes.

Intermittent Service Issues with Tracking Page Creator

May <var data-var='date'>28</var>, <var data-var='time'>10:17</var> EDT Resolved - This incident has been resolved. All events have been processing as expected since the fix was implemented yesterday morning. May <var data-var='date'>27</var>, <var data-var='time'>10:39</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. A backlog of older events are being processed now and any new events that come into the platform will be processed immediately. May <var data-var='date'>27</var>, <var data-var='time'>09:59</var> EDT Investigating - We are currently investigating an issue. You may experience issues receiving events to our integrations during this time.

Degraded Integration Performance

Apr <var data-var='date'> 1</var>, <var data-var='time'>11:10</var> EDT Completed - The scheduled maintenance has been completed. Apr <var data-var='date'> 1</var>, <var data-var='time'>11:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>28</var>, <var data-var='time'>10:43</var> EDT Scheduled - On Monday, April 1st, 2024, at approximately 11am Eastern Time, for up to 10 min, you may experience interruptions or limited availability while we upgrade our Event Queueing system. This proactive measure is designed to continue improving the reliability, redundancy, and performance of events processed by Malomo. All events that occur during this maintenance period will be processed once the system is back online. We appreciate your understanding and if you have any questions please reach out to our Support team via email (help@gomalomo.com) or on our website.

Scheduled Maintenance

Feb <var data-var='date'>20</var>, <var data-var='time'>20:10</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>20</var>, <var data-var='time'>19:22</var> EST Investigating - We are currently investigating an issue. You may experience issues accessing our dashboard, viewing tracking pages and receiving events to our integrations during this time.

Degraded Performance

Platform outage

Dec <var data-var='date'> 8</var>, <var data-var='time'>17:26</var> EST Resolved - We identified the issue causing some outgoing events to be delayed, lost, or duplicated. We implemented a fix on Dec. 6 at approximately 7:00 pm EST. We have monitored the fix for several days and have confirmed that we are not seeing any further issues. This issue is now resolved. Nov <var data-var='date'>29</var>, <var data-var='time'>22:19</var> EST Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>29</var>, <var data-var='time'>20:22</var> EST Update - We are continuing to work on a fix for this issue. Nov <var data-var='date'>29</var>, <var data-var='time'>18:34</var> EST Identified - Our team has identified an issue, affecting some merchants, that is impacting the timely delivery of order confirmation notifications from Malomo. This issue is causing delays in both email and SMS confirmations for orders processed through Malomo and its integrated services. We understand the importance of timely order updates and apologize for any inconvenience this may cause. Our technical team is actively working to resolve this issue as quickly as possible. As a temporary workaround we recommend affected merchants switch their order confirmation emails back to Shopify's default or use the Place Order metric in Klaviyo.

Delayed/missing notifications from Malomo

Nov <var data-var='date'>17</var>, <var data-var='time'>01:10</var> EST Resolved - This incident has been resolved. Nov <var data-var='date'>16</var>, <var data-var='time'>18:13</var> EST Monitoring - A fix has been implemented and we are monitoring the results. Nov <var data-var='date'>16</var>, <var data-var='time'>15:14</var> EST Investigating - We are currently investigating an issue. You may experience issues accessing our dashboard, viewing tracking pages and receiving events to our integrations during this time.

Oct <var data-var='date'>12</var>, <var data-var='time'>08:00</var> EDT Resolved - Starting at 8:00 AM EST we experienced an issue with a database that caused some slow down when processing events. The issue was resolved at 10:30 AM EST. There may be a 30 min. to 1 hour delay for events that were sent to our Attentive, Klaviyo and PostScript integrations.

Jul <var data-var='date'>24</var>, <var data-var='time'>15:38</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'>24</var>, <var data-var='time'>11:02</var> EDT Monitoring - At 10:10pm EDT, we implemented a fix and are currently monitoring the situation. Shopify orders that were placed between Friday, July 21 at 4:22pm EDT and Sunday, July 23 at 10:10pm EDT will see delayed Order Confirmation and Shipment Created updates as our system catches up. Orders placed after after July 23 at 10:10pm EDT will see immediate updates, as expected. Our team will continue to monitor the situation over the next 24 hours. Jul <var data-var='date'>23</var>, <var data-var='time'>21:58</var> EDT Investigating - This evening, we identified an interruption to our Shopify integration beginning Friday, July 21 at approximately 4:22pm EDT, which impacted events received from Shopify. Any orders placed after this time are being impacted in the following ways: • The Malomo: ShopifyOrderCreated metric is not being sent to our app integrations • The Malomo: ShipmentCreated metric is not being sent to our app integrations • Orders are not appearing in the Malomo dashboard If you trigger Order Confirmation emails or SMS from the Malomo: ShopifyOrderCreated metric in any integration, these order notifications likely have not been sent. If you trigger Just Shipped emails or SMS from the Malomo: ShipmentCreated metric in any integration, these shipping notifications likely have not been sent. We are currently investigating the issue and will provide updates as we learn more.

Delayed Order Created & Shipment Created Events

May <var data-var='date'>30</var>, <var data-var='time'>10:22</var> EDT Resolved - This incident has been resolved. May <var data-var='date'>25</var>, <var data-var='time'>10:18</var> EDT Monitoring - We identified an issue with our Klaviyo integration that could cause "ShipmentUpdateReceived" events with a status of "in_transit" to trigger multiple emails. A fix was introduced and we are now monitoring the situation. It's possible that one additional email may be triggered if the previous "ShipmentUpdateReceived" event with a status of "in_transit" occurred before the fix was implemented.

"ShipmentUpdateReceived" events triggering multiple emails in Klaviyo

May <var data-var='date'>23</var>, <var data-var='time'>07:05</var> EDT Resolved - This incident has been resolved. May <var data-var='date'>22</var>, <var data-var='time'>19:12</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>22</var>, <var data-var='time'>11:02</var> EDT Investigating - We are investigating reports of degraded performance and elevated error rates.

Degraded performance

Apr <var data-var='date'>17</var>, <var data-var='time'>21:12</var> EDT Resolved - This incident has been resolved. Apr <var data-var='date'>17</var>, <var data-var='time'>19:58</var> EDT Investigating - We are currently investigating this issue.

Mar <var data-var='date'> 7</var>, <var data-var='time'>11:22</var> EST Resolved - This incident has been resolved. Mar <var data-var='date'> 7</var>, <var data-var='time'>10:54</var> EST Investigating - We are currently investigating this issue.

Feb <var data-var='date'>10</var>, <var data-var='time'>09:45</var> EST Resolved - After monitoring this issue we have determined that the problem is fixed. USPS packages should now be correctly marked as "delivered" instead of "return-to-sender". Feb <var data-var='date'>10</var>, <var data-var='time'>09:41</var> EST Update - We are continuing to monitor for any further issues. Feb <var data-var='date'> 7</var>, <var data-var='time'>12:25</var> EST Monitoring - We have identified the issue and implemented a solution. We will continue to monitor the situation throughout the day. Feb <var data-var='date'> 7</var>, <var data-var='time'>11:32</var> EST Update - We are continuing to investigate this issue. Feb <var data-var='date'> 7</var>, <var data-var='time'>09:00</var> EST Investigating - Yesterday, we identified an issue with USPS marking some packages as “Return to Sender”, right before marking the same package as “Delivered”. We are currently working with our integration partner to determine why they are sending this event for packages that have been successfully delivered. Customers using Return to Sender flows with our integrations may be affected, causing some shoppers to receive Return to Sender notifications for USPS packages that were successfully delivered. We recommend that customers using Return to Sender flows disable them until the issue is resolved. Customers using the Problem Orders feature may see higher than average Returned to Sender numbers on the Problem Orders report. We will keep this post updated as we work to resolve the issue.

USPS incorrectly marking packages as “Return to Sender”

Oct <var data-var='date'>25</var>, <var data-var='time'>11:01</var> EDT Resolved - We have verified that the fix put in place yesterday (October 24) at 4:30pm ET has resolved this issue, and all delayed events have finished processing. Oct <var data-var='date'>24</var>, <var data-var='time'>18:55</var> EDT Monitoring - Earlier today, we identified an interruption to our Shopify integration beginning at 10:40am ET which impacted events received from Shopify. Our Malomo: ShopifyOrderCreated metric was not being sent to our app integrations as expected. If you trigger Klaviyo email or SMS notifications off the Malomo: ShopifyOrderCreated or Malomo: ShipmentCreated metrics, you were likely affected by this interruption. At 4:30pm ET, we implemented a fix and are currently monitoring the situation. Shopify orders that were placed today (October 24) between 10:40am ET and 4:30pm ET will see delayed Order Confirmation updates as our system catches up this evening. Orders placed after after 4:30pm ET will see immediate updates, as expected. Our team will continue to monitor the situation over the next 24 hours.

Is Malomo down?

Malomo status is UP

Lastest published events

Klaviyo event delivery failures

Tracking update issues

Delayed events

Malomo Services, Components and Regions

Recent Malomo events

Klaviyo event delivery failures

Tracking update issues

Delayed events

Klaviyo Integration Delayed Event Processing

Duplicate Order Created Events

Dashboard Orders List