Jul <var data-var='date'> 2</var>, <var data-var='time'>00:00</var> BST Resolved - This incident has been resolved. Jul <var data-var='date'> 1</var>, <var data-var='time'>21:19</var> BST Identified - We've identified the root cause and applied mitigations; service has recovered. We're continuing to monitor and are working on a permanent fix. Jul <var data-var='date'> 1</var>, <var data-var='time'>18:56</var> BST Investigating - We're experiencing elevated response times and error rates in our API/SDK endpoints. We're currently investigating.

Elevated API/SDK response times and error rates

Oct <var data-var='date'> 9</var>, <var data-var='time'>20:40</var> BST Resolved - The backlog of webhooks has been cleared. This incident has been resolved. Oct <var data-var='date'> 9</var>, <var data-var='time'>17:48</var> BST Identified - We've identified the issue. Webhooks are being accepted again. We're working through the backlog of webhooks now. Oct <var data-var='date'> 9</var>, <var data-var='time'>17:25</var> BST Investigating - We're currently experiencing delays in data processing and high error rate in incoming webhooks. Our team is investigating.

Delays in data processing

Sep <var data-var='date'>24</var>, <var data-var='time'>10:44</var> BST Resolved - This incident has been resolved. Sep <var data-var='date'>24</var>, <var data-var='time'>10:43</var> BST Update - We are continuing to investigate this issue. Sep <var data-var='date'>24</var>, <var data-var='time'>10:00</var> BST Investigating - The LoyaltyLion SDK was unavailable for 10 minutes

SDK unavailable

Aug <var data-var='date'>20</var>, <var data-var='time'>16:35</var> BST Resolved - This incident has been resolved. Aug <var data-var='date'>20</var>, <var data-var='time'>16:02</var> BST Monitoring - A fix has been implemented and we are now monitoring the results. Aug <var data-var='date'>20</var>, <var data-var='time'>15:08</var> BST Identified - We're currently experiencing elevated response times on our API endpoints. We have identified the issue and should have a fix deployed in the next hour.

High response time on API endpoints

Oct <var data-var='date'>29</var>, <var data-var='time'>09:58</var> GMT Resolved - This incident has been resolved. All our systems are operational and the data backlog has been cleared. Oct <var data-var='date'>28</var>, <var data-var='time'>23:27</var> GMT Monitoring - A fix has been implemented in our caching layer. We continue to monitor the results. Oct <var data-var='date'>28</var>, <var data-var='time'>20:27</var> GMT Update - Our systems remain stable. We are still working through the data processing backlog whilst continuing to monitor. Oct <var data-var='date'>28</var>, <var data-var='time'>17:49</var> GMT Identified - An issue in our caching layer led to elevated error rates and response times. We have identified the issue and stabilised our systems. We're now working through the data processing backlog while monitoring.

Partial API outage and delays in data processing

Aug <var data-var='date'>19</var>, <var data-var='time'>18:39</var> BST Resolved - This incident has been resolved. Aug <var data-var='date'>19</var>, <var data-var='time'>17:21</var> BST Investigating - We are currently investigating an issue with delays in data processing, this could cause delays to tier evaluations.

Mar <var data-var='date'> 4</var>, <var data-var='time'>02:00</var> GMT Resolved - Our SDK components went offline due to an issue with our CDN provider. We migrated to our backup CDN and full functionality was restored. This only impacted shopper-facing functionality (i.e. claiming rewards on-site), no API or backend requests were impacted

SDK outage

Mar <var data-var='date'> 2</var>, <var data-var='time'>13:30</var> GMT Resolved - We observed a performance degradation in one of our job queueing subsystems. As part of recovery to normal operations, we - paused processing on customer tier evaluation between 2024-03-02T13:30 UTC and 2024-03-02T19:00 UTC - paused processing on outbound webhook delivery between 2024-03-02T14:30 UTC and 2024-03-03T14:30 UTC - implemented controls on unusual traffic from Shopify Tier assignment and webhook delivery are now functioning as normal. We plan to gradually deliver the webhooks from the paused period over the next few days.

Job queue partial outage

Feb <var data-var='date'>20</var>, <var data-var='time'>21:49</var> GMT Resolved - This incident has been resolved. Feb <var data-var='date'>20</var>, <var data-var='time'>21:49</var> GMT Monitoring - The situation has been resolved, we're continuing to monitor Feb <var data-var='date'>20</var>, <var data-var='time'>21:16</var> GMT Identified - We've taken steps to triage and performance is stabilising. Feb <var data-var='date'>20</var>, <var data-var='time'>21:15</var> GMT Investigating - We're investigating increased API and platform latency. The merchant account is unaffected.

Increased API and platform latency

Jan <var data-var='date'>25</var>, <var data-var='time'>20:31</var> GMT Resolved - We've caught up with the backlog of jobs and our platform is back to normal. Jan <var data-var='date'>25</var>, <var data-var='time'>19:06</var> GMT Identified - Our data processing infrastructure is running behind which is causing delays data synchronisation and events such as email sending, customer search and integrations. We've identified the issue and we're working on it.

Data Processing Delays

Dec <var data-var='date'> 8</var>, <var data-var='time'>09:03</var> GMT Resolved - This incident has been resolved. Dec <var data-var='date'> 7</var>, <var data-var='time'>16:53</var> GMT Identified - Our background job processing infrastructure is running behind which is causing delays in events such us email sending, customer search and integrations. We've identified the issue and we're working on it.

Nov <var data-var='date'>29</var>, <var data-var='time'>11:00</var> GMT Resolved - Our background job processing infrastructure is running behind which is causing delays in events such us email sending and integrations. We've identified and addressed the problem, and we're working through the backlog of jobs.

Nov <var data-var='date'>24</var>, <var data-var='time'>16:35</var> GMT Resolved - This incident has been resolved. Nov <var data-var='date'>24</var>, <var data-var='time'>16:30</var> GMT Update - We're further increasing capacity to account for additional traffic over the BFCM period Nov <var data-var='date'>24</var>, <var data-var='time'>16:12</var> GMT Investigating - We are currently investigating this issue.

Partial API and platform outage

Nov <var data-var='date'> 8</var>, <var data-var='time'>20:52</var> GMT Resolved - A misconfigured cache caused elevated response times for our platform and API routes.

Elevated platform and API response time

Sep <var data-var='date'>19</var>, <var data-var='time'>17:16</var> BST Resolved - This incident has been resolved. Sep <var data-var='date'>19</var>, <var data-var='time'>17:11</var> BST Monitoring - We've identified the issue and applied a fix. We're currently monitoring to make sure our systems are fully recovered. Sep <var data-var='date'>19</var>, <var data-var='time'>16:52</var> BST Investigating - We are currently investigating elevated response time in our API and platform.

Elevated platform response time

Sep <var data-var='date'>13</var>, <var data-var='time'>21:19</var> BST Resolved - All systems are back to normal. We'll continue to monitor and perform a root cause analysis. Sep <var data-var='date'>13</var>, <var data-var='time'>20:48</var> BST Update - Most functionality has been restored. We're still working on a few background services. Sep <var data-var='date'>13</var>, <var data-var='time'>18:48</var> BST Update - Shopper-facing functionality has been restored. We're continuing our investigation in order to restore all services. Sep <var data-var='date'>13</var>, <var data-var='time'>17:34</var> BST Investigating - We are currently investigating elevated response time in our API and platform.

Aug <var data-var='date'> 2</var>, <var data-var='time'>20:55</var> BST Resolved - The situation has now been resolved Aug <var data-var='date'> 2</var>, <var data-var='time'>20:30</var> BST Update - We are continuing to investigate this issue. Aug <var data-var='date'> 2</var>, <var data-var='time'>20:30</var> BST Investigating - We're currently investigating elevated error responses from the LoyaltyLion API

Partial API outage

May <var data-var='date'> 1</var>, <var data-var='time'>23:58</var> BST Resolved - We saw an outage beginning at 23:58 BST on May 1st. Service was partially restored at 00:34 BST May 2nd and normal service levels resumed at 01:38 BST. We're still investigating the root cause of the outage and will update this incident as we learn more.

API and platform outage

May <var data-var='date'> 1</var>, <var data-var='time'>18:00</var> BST Resolved - Between 17:10-17:20 API and Platform endpoints experienced a partial outage. Some API requests may have failed during this period. By 17:20 normal service was restored.

Feb <var data-var='date'>22</var>, <var data-var='time'>13:15</var> GMT Resolved - Shopify merchants unable to sign in for ~90 minutes; no shopper-facing functionality was impacted

Merchant account sign in issues

Apr <var data-var='date'>27</var>, <var data-var='time'>13:48</var> BST Resolved - This incident has been resolved. Apr <var data-var='date'>27</var>, <var data-var='time'>12:29</var> BST Identified - The issue has been identified and a fix is being implemented.

Apr <var data-var='date'> 5</var>, <var data-var='time'>21:01</var> BST Resolved - This incident has been resolved. Apr <var data-var='date'> 5</var>, <var data-var='time'>20:21</var> BST Identified - The issue has been identified and a fix is being implemented.

Jul <var data-var='date'>15</var>, <var data-var='time'>04:00</var> BST Resolved - We are currently investigating this issue.

Mar <var data-var='date'>31</var>, <var data-var='time'>14:01</var> BST Resolved - This incident has been resolved. Mar <var data-var='date'>31</var>, <var data-var='time'>13:31</var> BST Monitoring - We've rolled out our fix and are monitoring the results Mar <var data-var='date'>31</var>, <var data-var='time'>12:36</var> BST Update - We are continuing to work on a fix for this issue. Mar <var data-var='date'>31</var>, <var data-var='time'>12:35</var> BST Update - We are continuing to work on a fix for this issue. Mar <var data-var='date'>30</var>, <var data-var='time'>01:45</var> BST Identified - We've identified the root cause of the issue and are working on a fix. Mar <var data-var='date'>30</var>, <var data-var='time'>01:15</var> BST Investigating - We're currently experiencing degraded performance on the merchant account; load times for both customer facing functionality and aggregate information in the merchant account are elevated.

Degraded merchant account & API performance

Mar <var data-var='date'>19</var>, <var data-var='time'>14:11</var> GMT Resolved - We've caught up with our backlog and all delayed jobs have now been processed. Mar <var data-var='date'>19</var>, <var data-var='time'>13:38</var> GMT Monitoring - We've just rolled out a fix and have resumed background job processing; we'll give another update when we've cleared the backlog. Mar <var data-var='date'>19</var>, <var data-var='time'>13:17</var> GMT Identified - We've identified the root cause of the issue; we've triaged the problem and are working on a permanent fix. Mar <var data-var='date'>19</var>, <var data-var='time'>12:50</var> GMT Investigating - We're currently investigating an issue relating to our background job processing. You may see delays in the following areas: * Customer email delivery * Launching loyalty tiers * Imports and exports Core customer-facing functionality including earning points, claiming rewards, and customer UI is unaffected.

Is LoyaltyLion down?

LoyaltyLion status is UP

Lastest published events

Elevated API/SDK response times and error rates

Delays in data processing

SDK unavailable

LoyaltyLion Services, Components and Regions

Recent LoyaltyLion events

Elevated API/SDK response times and error rates

Delays in data processing

SDK unavailable

High response time on API endpoints

Partial API outage and delays in data processing

Delays in data processing

Get your FREE account now,
5 monitors included!

Is LoyaltyLion down?

LoyaltyLion status is UP

Lastest published events

Elevated API/SDK response times and error rates

Delays in data processing

SDK unavailable

LoyaltyLion Services, Components and Regions

Recent LoyaltyLion events

Elevated API/SDK response times and error rates

Delays in data processing

SDK unavailable

High response time on API endpoints

Partial API outage and delays in data processing

Delays in data processing

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!