Sep <var data-var='date'> 2</var>, <var data-var='time'>01:39</var> CEST Resolved - An issue affecting collection synchronisation in Merchandising Cloud for some Neto and Norce customers. Some collections may be missing, along with associated product pins and links to merchandising rules or promotions. Our Engineering team is investigating the cause and assessing the impact.

Collection Synchronisation Issue

Jun <var data-var='date'>13</var>, <var data-var='time'>04:24</var> CEST Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>23:15</var> CEST Monitoring - Majority of our services are recovered / functioning as expected. We are continuing to closely monitor the broader outage affecting Google Cloud and Cloudflare. Jun <var data-var='date'>12</var>, <var data-var='time'>21:22</var> CEST Investigating - https://status.cloud.google.com/ https://www.cloudflarestatus.com/ We are currently observing partial service disruption due to a broader incident involving Google Cloud and Cloudflare infrastructure providers. This may result in degraded performance or limited accessibility for some users. Our team is actively monitoring the situation and working to mitigate any impact. We will provide updates as more information becomes available from the affected providers.

Google Cloud and Cloudflare Incident - Some Services Affected

Apr <var data-var='date'> 5</var>, <var data-var='time'>18:22</var> CEST Resolved - Continued monitoring to ensure smooth resolution for all merchants. Apr <var data-var='date'> 5</var>, <var data-var='time'>17:44</var> CEST Monitoring - The issue has been identified and affected merchants contacted. Apr <var data-var='date'> 5</var>, <var data-var='time'>14:01</var> CEST Update - There is a issue accessing the Shopify access token affecting some merchants Apr <var data-var='date'> 5</var>, <var data-var='time'>13:45</var> CEST Investigating - We are currently investigating this issue.

Shopify Access Token

Jan <var data-var='date'>24</var>, <var data-var='time'>13:33</var> CET Resolved - The incident have now been fully resolved. Actions are taken to avoid similar issues. Jan <var data-var='date'>24</var>, <var data-var='time'>10:31</var> CET Monitoring - Sync running as normal Jan <var data-var='date'>24</var>, <var data-var='time'>09:19</var> CET Identified - The issue has been identified and a fix is being implemented.

Pulsar experiencing issues causing delays in Sync & Analytics

Jan <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> CET Resolved - Norce has resolved the issue and product sync is now running again. Jan <var data-var='date'> 8</var>, <var data-var='time'>09:34</var> CET Identified - Norce Support team are working on identifying the issue and restoring the Hyperdrive access

Hyperdrive issue affects Norce Product Sync

Dec <var data-var='date'> 8</var>, <var data-var='time'>09:49</var> CET Resolved - Some clients experience delay in sync updates. The problem has been identified to a memory profile in the sync infrastructure & resolved. The memory has been increased & alarm implemented to prevent future problems. The search API has been unaffected and search, collections & recommendation results has been continuously served.

Temporary Delay in Sync

Nov <var data-var='date'>15</var>, <var data-var='time'>09:33</var> CET Resolved - The issue has been resolved by Norce Team. Nov <var data-var='date'>15</var>, <var data-var='time'>09:16</var> CET Investigating - All Norce hyperdrive feeds are returning errors and can hence not be downloaded/synced. The Norce team is investigating

Norce Hyperdrive Error

Oct <var data-var='date'> 3</var>, <var data-var='time'>15:41</var> CEST Resolved - An update has been released and the issue is resolved Oct <var data-var='date'> 3</var>, <var data-var='time'>14:53</var> CEST Identified - A few merchants running on custom feed sync is currently experience problems to update products. The root cause has been identified and the fix is being tested.

Partial issues with custom Sync

Jul <var data-var='date'>20</var>, <var data-var='time'>05:00</var> CEST Resolved - A subset of our merchants experienced interruptions in product sync. All other components of Findify were working as intended. The issue was identified and resolved.

Product Sync Affected Partially

Jun <var data-var='date'>15</var>, <var data-var='time'>17:00</var> CEST Resolved - In the afternoon of the 15th June 2023, we had an incident which caused partial outages in the US region. This was caused by an outburst of search requests which overloaded our Redis Connections. The issue was automatically detected by our internal alarming & immediately resolved where full recovery took 30-60 minutes to fully apply. No components were marked as affected by this incident. (.)

Partial outages in the US region

May <var data-var='date'>29</var>, <var data-var='time'>10:46</var> CEST Resolved - We are experiencing performance issues with Nodes in one cluster and a few clients are affected. Clients have been moved to functional clusters and the team are identifying the root cause.

Cluster Degraded Performance

May <var data-var='date'>11</var>, <var data-var='time'>10:48</var> CEST Resolved - At 10:47 we published an updated internal URL containing additional data for new functionality. This change was reviewed, tested & validated in our staging environment without any problems. Once released, our internal alarming immediately notified that clients were affected. The change was immediately rolled back but took a few minutes to take effect. At 11:03 service was confirmed to be back up running. We take this very seriously and are investigating why the initial tests didn't correctly reflect the outcome. Until that is resolved we will not publish any updates at all.

API Disrupted

Dec <var data-var='date'> 8</var>, <var data-var='time'>12:49</var> CET Resolved - 22:09 An error in Redis are affecting authentication. 22:14 The issue has been identified and a fix is being implemented. 22:19 80% restored 22:23 100 % restored. 22:28 The incident is validated to be fully restored. Monitoring is continued.

Redis Error affecting authentication

Nov <var data-var='date'>25</var>, <var data-var='time'>17:30</var> CET Resolved - During high volume inflow we experienced delays in the autoscaler which lead to outage of our main load balancer. This resulted in 25min of index disruptance and failure to readily serve product results. The team has scaled all systems manually and all functions are operational. We are further investigating why the autoscaler failed.

Autoscaling Deficiency

Sep <var data-var='date'>14</var>, <var data-var='time'>16:42</var> CEST Resolved - 16:42 Since a few minutes back we have an issue with indexing. Some clients are affected where indexing is not happening for the whole product catalogue. We are investigating this issue 16:54 The issue has been identified and a fix is being implemented. 16:57 A fix has been implemented and we are monitoring the results. 17:32 This incident has been validated to be resolved.

Index not fully updated

Jun <var data-var='date'>28</var>, <var data-var='time'>14:02</var> CEST Resolved - Today in the European morning (North American night) we had an issue in the access control layer, which affected the search api causing it to lose access to the configuration service. The team jumped on the issue immediately and it was solved within 20 minutes during which the service was partially unavailable. We are changing the access control method, to make it less error prone and more resilient in the future.

Search API - Access control layer

May <var data-var='date'>31</var>, <var data-var='time'>20:28</var> CEST Resolved - We just experienced a huge traffic spike (larger than we get on black Friday) that for a few minutes resulted in an increased error rate. We have increased capacity in the affected components, fine tuned the autoscaling and will now look into the source of the traffic to understand if we need to prepare for similar or even larger spikes going forward.

Depreciated search service

May <var data-var='date'>16</var>, <var data-var='time'>15:50</var> CEST Resolved - 15:50 Some clients are experiencing an increase error rate. We are looking into the root cause 15:53 This incident has been resolved.

Increased Error rate Search

May <var data-var='date'>10</var>, <var data-var='time'>11:12</var> CEST Resolved - 11:12 We are currently experiencing an issue with the admin API. This may prevent users to log into the dashboard. 11:19 The issue has been identified and a fix is being implemented. 12:42 A fix has been implemented and we are monitoring the results. 13:06 This incident has been resolved.

Access to Admin API

Jan <var data-var='date'>18</var>, <var data-var='time'>21:16</var> CET Resolved - 21:16 We have a service degradation on a number of clients. We are working to identify the root cause. 21:29 This incident has been resolved. We will continue to monitor the performance.

Service degradation search

Dec <var data-var='date'>23</var>, <var data-var='time'>10:02</var> CET Resolved - 10:02 We currently have an issue with the service that handles Findify analytics. For now Findify is unable to display some analytics in the merchant dashboard. The data is still available and we are working to have it restored. This does not affect the core functionality of search, collections and recommendations on site. 10:36 The issue has been resolved.

Analytics are not loaded for some of the merchant in the Merchant Dashboard

Dec <var data-var='date'>13</var>, <var data-var='time'>15:54</var> CET Resolved - We currently have an issue with the service that handles Findify analytics. For now Findify is unable to display any analytics in the merchant dashboard. There data is still available and we are working to have it restored. This does not affect the core functionality of search, collections and recommendations on site. Update: The issue was resolved over the weekend and the data is now restored.

Is Findify by Maropost down?

Findify by Maropost status is UP

Lastest published events

Collection Synchronisation Issue

Google Cloud and Cloudflare Incident - Some Services Affected

Shopify Access Token

Findify by Maropost Services, Components and Regions

Recent Findify by Maropost events

Collection Synchronisation Issue

Google Cloud and Cloudflare Incident - Some Services Affected

Shopify Access Token

Pulsar experiencing issues causing delays in Sync & Analytics

Hyperdrive issue affects Norce Product Sync

Temporary Delay in Sync