Jan <var data-var='date'> 2</var>, <var data-var='time'>16:00</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'> 2</var>, <var data-var='time'>15:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>31</var>, <var data-var='time'>18:12</var> UTC Scheduled - To ensure the continued stability and security of our systems, we will be upgrading our production Kubernetes version during the above time window. Impact: While we aim to complete this upgrade with minimal disruption, there is a possibility of intermittent service interruptions during the maintenance period. Action Required: No action is required on your part. Our team will be monitoring the process closely to minimize any potential downtime.

Scheduled Maintenance: Kubernetes Upgrade

Mar <var data-var='date'>10</var>, <var data-var='time'>16:00</var> UTC Resolved - Duration: 3/10 from ~9:00 am - 10:00 am PT (~1 hour) Issue: Starting at ~9:00 am PT on 3/10, some merchant builds failed due to 500 responses when fetching navigation data from the Storefront API. The navigation data service had encountered scaling issues that resulted in a lack of enough replicas being available for the volume of requests. Solution: By ~10:00 am PT, existing self-healing services had allowed navigation data service to stabilize and handle all incoming traffic. As a long-term solution, the number of minimum replicas enabled for the navigation data service has been doubled, and additional monitoring alerts have been implemented for this service.

Navigation Data Service Outage

Jan <var data-var='date'>26</var>, <var data-var='time'>04:00</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>26</var>, <var data-var='time'>03:02</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>24</var>, <var data-var='time'>18:29</var> UTC Scheduled - We will be undergoing scheduled maintenance to upgrade our Kubernetes cluster at 10pm ET on 1/25/2024. No downtime is expected during this period. The maintenance period should last an hour.

Nov <var data-var='date'>26</var>, <var data-var='time'>23:30</var> UTC Resolved - Nacelle experienced an outage to its Storefront API that impacted all merchants between 3:26 PM PT and 3:48 PM PT. This outage is currently under further investigation.

Storefront API Outage 11/26/2023

Nov <var data-var='date'>11</var>, <var data-var='time'>23:30</var> UTC Resolved - Between 6:26 PM ET and 7:21 PM ET, Nacelle experienced a spike of new requests that required Nacelle’s DevOps team to scale additional Kubernetes pods. While these new pods were scaling up, the API experienced degraded performance (e.g., long latencies). The root cause of the issue was how Nacelle's custom schema fetching works across new additions to our GraphQL interface. To ensure stability over Black Friday / Cyber Monday, the Nacelle team has removed features still in development from the Storefront API schema service.

Degraded Storefront API Performance

Jul <var data-var='date'> 5</var>, <var data-var='time'>21:40</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'> 5</var>, <var data-var='time'>21:02</var> UTC Identified - We've narrowed this issue down to a handful of spaces. Further investigation is ongoing. We've downgraded the impact of this incident based on these findings. Jul <var data-var='date'> 5</var>, <var data-var='time'>20:18</var> UTC Investigating - We're currently investigating an outage related to collection reindexing. As we continue to investigate, we recommend _not_ reindexing to ensure missing collection data does not impact your storefront.

Collections Index Outage

Jun <var data-var='date'>28</var>, <var data-var='time'>19:54</var> UTC Resolved - This issue is now resolved. Webhook updates are now coming through as expected. Jun <var data-var='date'>28</var>, <var data-var='time'>18:29</var> UTC Investigating - We're currently experiencing intermittent failures for Shopify webhook product/update events. This is only impacting merchants who are using the 2023-01 version of the Shopify API. We're currently investigating and will provide updates as soon as possible. As we work to remedy this issue, please use manual space reindexes to solve any data consistency problems.

Degraded Shopify Webhook Performance

Mar <var data-var='date'>22</var>, <var data-var='time'>20:48</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>22</var>, <var data-var='time'>19:17</var> UTC Investigating - We are investigating an issue impacting Contentful webhooks. Users looking to index data from Contentful should manual re-index within the Nacelle dashboard.

Contentful Webhooks Down

Dec <var data-var='date'>31</var>, <var data-var='time'>02:25</var> UTC Resolved - This incident has been resolved. Dec <var data-var='date'>30</var>, <var data-var='time'>23:56</var> UTC Identified - The issue has been identified and a fix is being implemented. Dec <var data-var='date'>30</var>, <var data-var='time'>20:44</var> UTC Investigating - We're currently investigating an issue with V1 data ingestion. More information will be shared as soon as possible.

V1 Ingestion Performance Degraded

Nov <var data-var='date'> 4</var>, <var data-var='time'>00:30</var> UTC Resolved - From 5:40pm to 10:25pm PDT Nacelle was experiencing issues related to Storefront API “content” queries on the v2 platform. This could have had an effect on builds during this window and/or client side content calls.

Is Nacelle down?

Nacelle status is UP

Lastest published events

Scheduled Maintenance: Kubernetes Upgrade

Navigation Data Service Outage

Scheduled Maintenance: Kubernetes Upgrade

Nacelle Services, Components and Regions

Recent Nacelle events

Scheduled Maintenance: Kubernetes Upgrade

Navigation Data Service Outage

Scheduled Maintenance: Kubernetes Upgrade

Storefront API Outage 11/26/2023

Degraded Storefront API Performance

Collections Index Outage