Feb <var data-var='date'>12</var>, <var data-var='time'>07:00</var> UTC Resolved - A one-off backend job caused an overload on some of our backend data servers, causing the associated processes to crash. We promptly restarted the affected processes and they came back online. While these backend data servers were down, our front-end servers continued to serve requests using intermediate layers of caching, and most end users continued to receive recommendation results. After bringing the data servers online, we were able to resume running the one-off job with updated settings that did not put production infrastructure at risk. During the period that the affected processes were down, the following issues would have been observed: * Recent updates made (to the catalog and to ML data) would not be getting pulled in to recommendations. * Infrequently accessed data (that was not cached) would not be successfully looked up, which could cause issues such as not returning enough results (particularly for queries with restrictive rules), or not returning the best results. We had two periods where users may have noticed issues: * US East from 2025-02-11 10:59 PM PT (2025-02-12 06:59 UTC) to 2025-02-11 11:26 PM PT (2025-02-12 07:26 UTC), mostly affecting traffic from the Eastern United States and Canada, Europe, and South America * US West from 2025-02-11 11:38 PM PT (2025-02-12 07:38 UTC) to 2025-02-11 11:59 PM PT (2025-02-12 07:59 UTC), mostly affecting traffic from the Western United States and Canada, Asia, and Australia Our redundant architecture prevented visible damage to end users. We are investigating best practices around the right settings, configurations, and safeguards for one-off backend jobs to reduce the risk of these sorts of incidents occurring in the future.

Backend server overload resulting in issues with recommendation quality as well as update lags

May <var data-var='date'>19</var>, <var data-var='time'>12:16</var> UTC Resolved - This incident has been resolved.May <var data-var='date'>19</var>, <var data-var='time'>11:50</var> UTC Monitoring - An initial fix has been implemented and most of the affected processing has been restored, including the inventory API endpoints. We are working on residual fixes and monitoring the results.May <var data-var='date'>19</var>, <var data-var='time'>11:00</var> UTC Investigating - We are investigating issues with inventory API endpoints, which seems to have been mostly returning 503 error codes since 12:20 AM PT (2024-05-19T07:20:00Z). Updates to catalog (both insertions and deletions) are affected.

Issues with inventory API and related background processing

May <var data-var='date'>17</var>, <var data-var='time'>02:20</var> UTC Resolved - Google Cloud confirms having fully resolved the incident https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre on their end, and all our systems appear to be stable.May <var data-var='date'>17</var>, <var data-var='time'>00:05</var> UTC Monitoring - The email service is functioning normally -- so on-open email recommendations should resume working as expected. We are still seeing some symptoms of issues with Google Cloud's backend services, and are waiting on Google Cloud to address those issues and confirm resolution of the underlying issue at https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre. We will continue to monitor the situation closely until they have done so.May <var data-var='date'>16</var>, <var data-var='time'>23:51</var> UTC Identified - Google Cloud has posted details of the incident on their end: https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre. We are waiting on them to resolve the issue.May <var data-var='date'>16</var>, <var data-var='time'>23:25</var> UTC Investigating - We are investigating issues with our email rendering service due to which on-open email recommendations are expected to not render properly. This is likely due to issues with the Google Cloud infrastructure. We're still awaiting confirmation from Google Cloud that there's an issue on their end. In the meantime, we are looking into the best ways to mitigate and address the issue. Another infrastructure component of ours is also affected, but this component is not end-user-facing and will not have an immediate impact on the end user experience.

Issues with email service (and another component that is not end-user-facing) likely due to Google Cloud infrastructure issues

Jun <var data-var='date'> 4</var>, <var data-var='time'>15:25</var> UTC Resolved - Google Cloud has confirmed resolution on their end, and all alerts have resolved on our end. We're marking the incident as resolved. If you're a customer who is still seeing infrastructure issues, or if you have any questions about the incident, please reach out to us at liftignitersupport@thearenagroup.net.Jun <var data-var='date'> 4</var>, <var data-var='time'>15:11</var> UTC Monitoring - Capacity has been reprovisioned in the us-east1 region, and we resumed routing traffic to US East. While our metrics seem stable as of now, Google Cloud has not yet confirmed resolution of the issue on their end, so we will continue to monitor the situation for the time being.Jun <var data-var='date'> 4</var>, <var data-var='time'>14:25</var> UTC Identified - Due to a Google Cloud issue creating new VM instances in the us-east1 region (https://status.cloud.google.com/incidents/DyCcy7iGtWqLYEyJBWrC#7Nro3snGH5YXCyUfQuKS), we were having difficulty serving traffic that was being directed to that region. We were able to minimize the damage by directing as much traffic as possible to the us-west1 region that is still operational. We are monitoring for updates from Google Cloud as well as assessing the infrastructure impact.

Capacity issues in US East region due to Google Cloud infrastructure issues (they are unable to provision new instances)

Dec <var data-var='date'> 1</var>, <var data-var='time'>04:59</var> UTC Resolved - This incident has been resolved.Nov <var data-var='date'>30</var>, <var data-var='time'>23:22</var> UTC Monitoring - LiftIgniter console was taken down temporarily to update API keys. This may have caused some API calls to fail. Affected customers have been notified, and all systems should be back up and running.Nov <var data-var='date'>30</var>, <var data-var='time'>18:13</var> UTC Investigating - https://console.liftigniter.com has temporarily been taken down to investigate an issue. Inventory updates are also experiencing issues.

Issues with console and the API usage

Apr <var data-var='date'>28</var>, <var data-var='time'>21:33</var> UTC Resolved - Capacity is back to normal and all configurations have been returned to their defaults.Apr <var data-var='date'>28</var>, <var data-var='time'>18:21</var> UTC Monitoring - Due to some capacity issues being experienced by our cloud provider (Google Cloud) in US East, we are or were experiencing issues with some of our services. Our query endpoint (query.petametrics.com), that is used to serve recommendations, saw (503 status) error rates rise to about 1%. Error rates were nonzero between 18:00 and 18:04 UTC. We had already started provisioning alternate capacity prior to the increase in error rates, but still got some errors as the provisioning of capacity took a few minutes. We also saw increased latency in the period from 17:51 to 18:11 UTC for the successful requests. We also provisioned alternate capacity for a few other affected services; these services had a few minutes of downtime while the alternate capacity was coming online. We significantly benefited from preparation we did after the previous incident http://status.liftigniter.com/incidents/1522vrjxbmcp.

Issues with services in US East due to capacity issues with cloud provider

Apr <var data-var='date'> 9</var>, <var data-var='time'>00:59</var> UTC Resolved - Capacity is back to normal and all services are operating normally. We've identified improvements to make to our systems to make them even more robust to similar issues.Apr <var data-var='date'> 8</var>, <var data-var='time'>15:35</var> UTC Monitoring - All our services are back to working normally. We are still waiting for the underlying capacity issues to be fixed, and will be reviewing our setup to see how we can reduce the impact of such incidents in the future.Apr <var data-var='date'> 8</var>, <var data-var='time'>15:15</var> UTC Update - As of 15:04 UTC, our email-rendering services are back online and working properly, so all our front-facing services are working properly now. We have identified that the capacity issue is affecting one of our backend services used for managing user histories, and are continuing to investigate that.Apr <var data-var='date'> 8</var>, <var data-var='time'>14:50</var> UTC Identified - Due to some capacity issues being experienced by our cloud provider (Google Cloud) in US East, we are or were experiencing issues with some of our services. Our query endpoint (query.petametrics.com), that is used to serve recommendations, saw (503 status) error rates rise to over 1%, peaking at 2.6% briefly. Error rates were nonzero between 14:23 and 14:38 UTC. Error rates went down to zero after we provisioned alternate capacity. The period of increased error rates was also a period of increased latency for the successful requests. We are currently investigating the impact on some of our other services, including a service used for rendering emails.

Nov <var data-var='date'>16</var>, <var data-var='time'>22:30</var> UTC Resolved - Google has confirmed full resolution at https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh and we have reverted to our normal settings. All metrics have returned to normal ranges.Nov <var data-var='date'>16</var>, <var data-var='time'>18:21</var> UTC Monitoring - Starting around 18:10 UTC (10:10 AM Pacific Time) we started seeing the endpoints working again. We paused our mitigation steps but are continuing to monitor before reverting them. Google Cloud has posted an incident report at https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh and the incident report does not yet confirm resolution. The incident has also been reported in the media; see https://www.theverge.com/2021/11/16/22785599/google-cloud-outage-spotify-discord-snapchat-google-cloud for instance.Nov <var data-var='date'>16</var>, <var data-var='time'>18:08</var> UTC Identified - All our domains that route through Google Cloud's global public load balancer are giving 404 errors. This appears to be due to issues with Google Cloud; as of the time of writing this, they do not have an incident page but are reported degraded service on their status page (https://status.cloud.google.com/). We implemented mitigation procedures for our recommendation engine endpoints that we originally wrote after a 2018 incident http://status.liftigniter.com/incidents/7kz9f7w1z8jg and expect to mitigate the bulk of the impact through these mitigation procedures, even if Google Cloud takes time to solve its issues. However, some of our other services, including our inventory API, user API, email recommendations, and console do not have a similar mitigation process in place so recovery for them must wait till Google Cloud fixes the issue.

Google Cloud load balancer issues causing errors for end users; workarounds in progress for core services

Oct <var data-var='date'>21</var>, <var data-var='time'>14:35</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>21</var>, <var data-var='time'>14:34</var> UTC Investigating - For a little under an hour, from 2021-10-20 8:04 UTC to about 2021-10-20 8:46 UTC, MaxCDN, one of the CDN providers that we use to serve traffic, was slow and unresponsive for part of our Australia traffic due to scheduled maintenance in the Sydney region: https://status.maxcdn.com/incidents/0stcf5zpm766. Customers with significant web traffic volume in Australia were affected. Customers using a version of the LiftIgniter snippet that automatically tries against the Cloudfront CDN when MaxCDN is down woud be partly insulated. Even so, since there is some time lag for the retry, we did fail to capture data for their end users who bounced before the retry could be triggered, and failed to show recommendations for users who scrolled to recommendation areas before the retry had kicked in. API integrations were not affected, so apps were unaffected. We are working with MaxCDN's parent company, Stackpath, to transition from MaxCDN's infrastructure to Stackpath's infrastructure, which we expect to be more robust to scheduled maintenance. We expect such incidents to stop happening once we transition.

CDN issues with MaxCDN (one of our CDN providers)

Oct <var data-var='date'>12</var>, <var data-var='time'>08:00</var> UTC Resolved - For about an hour, from 2021-10-12 7:57 UTC to about 2021-10-12 8:55 UTC, MaxCDN, one of the CDN providers that we use to serve traffic, was slow and unresponsive for part of our Japan traffic due to scheduled maintenance in the Tokyo region: https://status.maxcdn.com/incidents/n6s35q0f8gf0. Customers with significant web traffic volume in Japan were affected. The majority of such customers by traffic are using a version of the LiftIgniter snippet that automatically tries against the Cloudfront CDN when MaxCDN is down, so they were partially insulated. Even so, since there is some time lag for the retry, we did fail to capture data for their end users who bounced before the retry could be triggered, and failed to show recommendations for users who scrolled to recommendation areas before the retry had kicked in. API integrations were not affected, so app customers in Japan were not affected.

Nov <var data-var='date'> 5</var>, <var data-var='time'>13:51</var> UTC Resolved - Data restoration is now complete, and all metrics are back to normal. The backup from which we restored is a little older than the point in time at which we lost data, so there is a possibility of some intermediate data loss; we are keeping a close eye on performance to see if there are data inconsistencies.Nov <var data-var='date'> 3</var>, <var data-var='time'>19:17</var> UTC Update - Capacity has been restored in the US East region; latency and other performance metrics are close to normal. However, we are still faced with the problem that the data available to the recommendation engine for queries in the US East region is less complete than it should be, causing degraded recommendation quality in some cases. We are working to restore from backups.Nov <var data-var='date'> 3</var>, <var data-var='time'>18:57</var> UTC Identified - We are currently experiencing hardware issues that affected multiple nodes of our datastore in the US East. These hardware issues are causing some data to be unavailable to our recommendation engine when calculating recommendations (causing a degraded quality of recommendations), and may also result in increased latencies when returning recommendations. We are working to restore capacity and will then restore lost data from backups. Our system in US West is not affected as of now. We will share more details as they become available.

Datastore issues in US East region causing degraded recommendation quality

Nov <var data-var='date'> 5</var>, <var data-var='time'>13:50</var> UTC Resolved - Data restoration is now complete and all metrics are back to normal. We will continue to keep an eye for any data inconsistencies created by the restore process, but nothing seems off as of now.Nov <var data-var='date'> 4</var>, <var data-var='time'>09:55</var> UTC Identified - This is similar to http://status.liftigniter.com/incidents/1z9fqwpckkyk We experienced a hardware issue affecting multiple nodes of out datastore in US West [EDIT: Our hosting provider, Google Cloud, believes that this was actually a software issue with their virtualization software, and not a real hardware issue]. Capacity has been restored (it was limited between 8:30 UTC and 9:41 UTC). During the period of limited capacity, we experienced increased latency and substantially degraded recommendation quality. Now that capacity has been restored, the ongoing challenge is that due to the large amount of node failures, the datastore system in US West lacks all the data that it should have, causing degraded recommendation quality in some cases. We are working to restore from backups.

Datastore issues in US West region causing degraded recommendation quality

Sep <var data-var='date'>11</var>, <var data-var='time'>16:14</var> UTC Resolved - We believe that the outage was caused by the load balancer incorrectly marking the service as unhealthy, due to aggressive timeout and failure conditions. This caused downtime for a brief while. We are addressing this by changing the failure thresholds for the load balancer to be more generous. EDIT: We determined that the initial marking as unhealthy was likely related to a live migration that our hosting provider, Google Cloud, conducted on a number of servers at the time.Sep <var data-var='date'>11</var>, <var data-var='time'>15:51</var> UTC Monitoring - The email service seems to be back up, as all alerts have resolved. We believe the likely cause was in the networking and load balancing layers, as the servers themselves were operational throughout the duration of the incident. However, we're continuing to investigate what happened.Sep <var data-var='date'>11</var>, <var data-var='time'>15:48</var> UTC Investigating - We are investigating issues with the email service that powers on-open email recommendations, based on alerts we received at 8:44 AM Pacific Time (15:44 UTC). The issues could be affecting new email opens. We will post more details once we have them.

Issues with email service that powers email recommmendations

Aug <var data-var='date'>13</var>, <var data-var='time'>08:30</var> UTC Resolved - On August 13, 2020, between 1:11 AM and 1:25 AM Pacific Time (PDT), we saw increased error rates and unresponsiveness on our model-server system in US West due to a mass killing of servers by Google Cloud, our hosting provider. The automatic response was sufficient to stabilize the system. No manual intervention was needed, though our engineers did get alerted and were monitoring the situation.

Brief period of increased error rates in US West model-servers due to a mass killing of servers by our hosting provider

Aug <var data-var='date'> 8</var>, <var data-var='time'>05:25</var> UTC Resolved - We have pushed robustifications to our serving architecture. We expect that with these robustifications, problems similar to the ones that triggered this incident would not cause internal server errors. Rather, such problems would trigger an alert in our system for exceptions, while still returning a possibly degraded response to the queries. The robustifications will be tested next week as we resume the work of pushing updates to our machine learning building configurations.Aug <var data-var='date'> 6</var>, <var data-var='time'>22:05</var> UTC Monitoring - We have successfully reverted the faulty code push and the internal server errors have stopped completely. We are working on robustifications on the serving side that will prevent it from throwing errors for similar corruptions on the building side.Aug <var data-var='date'> 6</var>, <var data-var='time'>21:40</var> UTC Identified - LiftIgniter's recommendation servers are experiencing a significant increase in internal server errors and returning status code 500. We identified a recent code push to our machine learning building configuration that is responsible, and are working on reverting. We will update with more details as we learn more.

Significant increase in 500 internal server errors in recommendations

Jul <var data-var='date'>23</var>, <var data-var='time'>20:15</var> UTC Resolved - This incident has been resolved.

Issue with honoring extremely short TTLs for items in the catalog

Jul <var data-var='date'>12</var>, <var data-var='time'>04:03</var> UTC Resolved - This incident has been resolved.Jun <var data-var='date'>30</var>, <var data-var='time'>17:09</var> UTC Monitoring - The backlog was fully caught up as of 2020-06-30 16:00 UTC (about one hour ago at the time of posting this). Recommendation-serving is therefore operating completely normally now. The customer website that was dramatically affected has been restored to its prior configuration of sending traffic to the closest region. Also, after removing the node with disk corruption, we have successfully rebalanced data across the remaining nodes, and our data storage is operating as expected.Jun <var data-var='date'>30</var>, <var data-var='time'>03:42</var> UTC Update - Our machines have started working through the backlog of catalog updates. We are working on getting an estimate of how quickly the backlog will be fully caught up.Jun <var data-var='date'>30</var>, <var data-var='time'>03:40</var> UTC Identified - LiftIgniter maintains servers to serve recommendations in two Google Cloud regions: US West and US East. End user traffic generally goes to the closer of the two regions unless configured to go to a specific region. The LiftIgniter recommendation servers in the US West region do not have access to catalog updates made since around 2020-06-29 21:20 UTC (about 6 hours ago at the time of posting this). We have determined that this is because of disk corruption on one of the nodes blocking updates from being made. No data is lost, and we are working to replace the bad node and resume the process of catalog updates. Affected customers are those whose traffic mainly goes in the US West region, and this primarily includes customers with end users in the Western United States and Asia Pacific. The main effect on these customers will be the lack of freshness of catalog information used to serve recommendations. Only one customer website was dramatically affected, and we are in touch with them. We've directed their end user traffic to the US East region for the time being.

Catalog updates backlogged for recomendation-serving in US West

Jun <var data-var='date'>22</var>, <var data-var='time'>13:56</var> UTC Resolved - On Friday, June 19, 2020, around 20:48 UTC, LiftIgniter released an updated version of the JS SDK file that fixes the unavailability of diagnostic and debugging commands (specifically, $p("runDiagnostics") and $p("printDebugInfo")) on IE 9, 10, and 11. With the new version, if it is found that the browser does not support window.Promise, the user is prompted to load a global promise polyfill using $p("loadPromise"), and then rerun the diagnostic command. This should allow users to access diagnostic and debugging functionality on IE 9, 10, and 11, where it was previously unavailable due to window.Promise not being present on these browsers. Customers who are loading their own global promise polyfill are unaffected; diagnostic and debugging commands would already have been available to them in IE 9, 10, and 11 prior to this fix.Jun <var data-var='date'> 9</var>, <var data-var='time'>15:02</var> UTC Monitoring - We have deployed the first fix to the JS SDK file (the fix was released at 7:18 AM PDT, or 14:18 UTC). The core functionality of the file should now work on IE 9, 10, and 11. Diagnostic and debugging commands are still not available. We have also communicated with the customers who reached out individually regarding this.Jun <var data-var='date'> 9</var>, <var data-var='time'>14:17</var> UTC Identified - On Sunday June 7, LiftIgniter released a new version of its JavaScript SDK file. The new version used a smaller file, with some diagnostic and debugging functionalities moved to separate chunk files. The logic used to handle these chunk files was not compatible with IE 11 and earlier versions of IE, so the script stopped working for these users. We have a hotfix ready and are rolling it out. The hotfix still does not provide diagnostic functionalities on IE 11 and below, but at least the core JS SDK works correctly on these browsers now.

JS SDK file not working on IE 11 since 11:30 AM PT (18:30 UTC) on Sunday June 7

Oct <var data-var='date'>23</var>, <var data-var='time'>04:23</var> UTC Resolved - The impact of the networking issue on LiftIgniter's services has been addressed. All traffic is now being routed normally. The recovery proceeded without incident (there were no latency or timeout alerts during the transition).Oct <var data-var='date'>23</var>, <var data-var='time'>01:08</var> UTC Identified - Due to a networking issue reported by Google Cloud at https://status.cloud.google.com/incident/cloud-networking/19020 all activity and requests traffic for the query.petametrics.com endpoint is being sent to US East rather than US West. This may mean increased latencies for users who are closer to the US West region. We had a brief period of increased error rates for a few minutes after the transition, which happened between 5:36 PM and 5:38 PM. However, our capacity in US East scaled up quickly to meet the traffic increase, so the errors stopped quickly within about 5 minutes.

All non-region-specific traffic being sent to US East instead of US West due to Google Cloud networking issues

Oct <var data-var='date'>18</var>, <var data-var='time'>00:44</var> UTC Resolved - We have deployed a code update that we expect will make the system more robust and the likelihood of this kind of downtime much lower. Even prior to this update, we haven't had this issue for the past 4 weeks. We're marking the incident as resolved.Sep <var data-var='date'>19</var>, <var data-var='time'>20:56</var> UTC Monitoring - We have just recovered from degraded performance for our inventory API servers used for our inventory API operations on the api.petametrics.com domain. We will share more details later, but the problems we noticed were: - Elevated rates of 5XX errors for insertion operations starting around September 19, 2019 1:20 PM Pacific Time (20:20 UTC) - Servers unresponsive to pings intermittently between on September 19, 2019 between 1:30 PM Pacific Time and 1:45 PM Pacific Time (20:30 to 20:45 UTC). We have reinstated server capacity and are currently reviewing the situation.

Inventory API servers giving errors and experiencing downtime

Aug <var data-var='date'>23</var>, <var data-var='time'>17:40</var> UTC Resolved - We have pushed some code updates that turn off the parts of the codebase that were causing trouble, and also added several more code robustness improvements, better alerting, and better playbooks for alert response. We expect that incidents with the same cause won't occur any more, and other incidents with similar symptoms will be mitigated much more quickly. We are still diagnosing the exact mechanism by which the problem occurred (so that we can reactivate the parts of the codebase we've turned off). We're also preparing an internal postmortem. We will share further details regarding timeline and impact once the investigation and internal postmortem are completed.Aug <var data-var='date'>22</var>, <var data-var='time'>13:10</var> UTC Monitoring - The inventory API servers had issues between August 21, 2019, 10:45 PM PDT and 11:45 PM PDT (August 22, 2019, 5:45 UTC to 6:45 UTC). A large fraction of requests to inventory insertion, GET, and DELETE operations timed out or gave error codes during the period. We were able to get the servers back to normal through scaling up capacity, and the servers have been stable since 11:45 PM PDT. Customers who received failures or timeouts on inventory API operations during this period would see their requests succeed if they retried after 11:45 PM PDT. We are still reviewing what happened and will update with more details later. NOTE: This degraded performance only affects inventory insertion, GET, and DELETE operations attempted via the API during the time period. Any affected users would have either had their request time out or received an error code. Customers who do not use the inventory API, or who were not using it during the time period of the problem, are unaffected.Aug <var data-var='date'>22</var>, <var data-var='time'>06:21</var> UTC Investigating - We are experiencing degraded performance for our inventory API servers used for our inventory API operations on the api.petametrics.com domain. The problems appear to have started a little before 11 PM PDT. We will post more details as we get to know them. NOTE: This degraded performance only affects inventory insertion, GET, and DELETE operations performed via the API. It has no direct effect on the model-servers that server queries. Therefore it only affects the users who use our inventory API, and specifically, used it during that time period.

Issues with our inventory API servers

Jul <var data-var='date'> 3</var>, <var data-var='time'>16:14</var> UTC Resolved - We are making this incident resolved after verifying that traffic is being distributed between US West and US East in the normal manner. We believe that our system's response to the networking issues was graceful and resulted in minimal end user impact from the outage. We will keep an eye out for further details from Google about the incident https://status.cloud.google.com/incident/cloud-networking/19015 to learn more.Jul <var data-var='date'> 3</var>, <var data-var='time'>13:51</var> UTC Monitoring - The Google Cloud load balancer has now resumed sending traffic to US East. Everything seems to be working as expected, but we are closely monitoring metrics and will resolve this incident once all the metrics look healthy for a while.Jul <var data-var='date'> 2</var>, <var data-var='time'>18:34</var> UTC Update - Although Google posted in https://status.cloud.google.com/incident/cloud-networking/19015 at 9:12 AM PDT (16:12 UTC) that the problem is fully resolved, we are continuing to see all our traffic being sent to US West. We have opened a case with Google to check in on the status. We continue to believe that none of our services are affected (except possibly for slight latency increases). We will post further updates once we hear back from Google or see that the load balancer is directing traffic to US East.Jul <var data-var='date'> 2</var>, <var data-var='time'>16:05</var> UTC Identified - On Tuesday, July 2, 2019, between 08:22 and 08:24 AM Pacific Time, which is 15:22 to 15:24 UTC, we saw a dramatic reduction in the traffic going to our US East datacenter and a corresponding increase in the traffic going to our US West datacenter. The change in traffic appears to be due to the Google Load Balancer, a global public load balancer provided by our cloud provider Google Cloud Platform, deciding to no longer direct traffic to US East. Our autoscaling was able to handle the approximate doubling of traffic to US West fairly gracefully, with capacity roughly doubling within minutes. We believe that Google Cloud's decision to redirect traffic is driven by networking issues with US East as described at https://status.cloud.google.com/incident/cloud-networking/19015 According to Google's update at 08:50 AM PDT (15:50 UTC): "The Cloud Networking service (Standard Tier) has lost multiple independent fiber links within us-east1 zone. Vendor has been notified and are currently investigating the issue. In order to restore service, we have reduced our network usage and prioritised customer workloads. We will provide another status update by Tuesday, 2019-07-02 09:38 US/Pacific with current details." We will await further updates from Google. As far as we can make out, there is no impact on the availability of LiftIgniter's services. Even the regional endpoint for US East (query-us-east1.petametrics.com) appears to be working correctly. However, there may be a small end-user latency impact, both for customers who have hardcoded query-us-east1.petametrics.com as the endpoint (due to the networking issues in US East) and to the customers whose end users would normally go to US East, but are now being redirected to the somewhat more distant US West.

Google Cloud networking incident in US East: Minimal impact on LiftIgniter service other than slight latency increases

Jun <var data-var='date'> 3</var>, <var data-var='time'>00:39</var> UTC Resolved - Google has confirmed at https://status.cloud.google.com/incident/cloud-networking/19009 that the networking issue is resolved and that they will post a detailed report. Since all our alerts have also resolved and our systems have been stable for the past few hours, we are marking the incident resolved as well.Jun <var data-var='date'> 2</var>, <var data-var='time'>21:05</var> UTC Monitoring - Our systems have been stable for 40 minutes now, but we are still waiting for Google to confirm in https://status.cloud.google.com/incident/cloud-networking/19009 that they have fixed the issue on their end before we consider this issue resolved.Jun <var data-var='date'> 2</var>, <var data-var='time'>20:17</var> UTC Update - The services have returned again to a fully functional state in both regions. However, we are still waiting for more details from Google Cloud regarding the networking issue at https://status.cloud.google.com/incident/cloud-networking/19009Jun <var data-var='date'> 2</var>, <var data-var='time'>20:06</var> UTC Identified - We noticed a recurrence of the problem in US West (previously, the problem had been more severe in US East) and are applying the same emergency fix on US West. We expect to return to fully functional status in 10 to 15 minutes. Also, Google Cloud has clarified at https://status.cloud.google.com/incident/cloud-networking/19003 that the issue is related to a larger networking issue (which is also what we originally saw evidence for). Their status page on the networking issue is at https://status.cloud.google.com/incident/cloud-networking/19009Jun <var data-var='date'> 2</var>, <var data-var='time'>19:59</var> UTC Monitoring - All our services appear to be fully functional again. However, we are still waiting for Google to share more details of the underlying issue at https://status.cloud.google.com/incident/compute/19003 so we can evaluate how much longer to closely monitor our systems and whether there may be any other impact missed by our alerts.Jun <var data-var='date'> 2</var>, <var data-var='time'>19:38</var> UTC Identified - Google Cloud has reported the issue with Google Compute Engine at https://status.cloud.google.com/incident/compute/19003 They appear to have recovered enough that we should be able to get our services to a fully functional state soon. However, because they continue to have degraded performance, we will keep an eye on the impact on our services.Jun <var data-var='date'> 2</var>, <var data-var='time'>19:02</var> UTC Investigating - We received alerts suggesting that our services in various regions are having trouble talking to each other as well as to external services. This is affecting the volume of traffic that is being successfully processed by all our endpoints under query.petametrics.com and api.petametrics.com and is also affecting the accessibility of the LiftIgniter Console. These networking issues may be due to our cloud provider. We are still investigating to mitigate the situation and assess the impact. EDIT: After more investigation we are more confident that the issues are due to our cloud provider, Google Cloud, but are still waiting for them to report the issues on their Status page https://status.cloud.google.com It looks like others have also noticed the same issues with Google Cloud; see for instance https://twitter.com/GossiTheDog/status/1135260263316381696 https://twitter.com/phineyes/status/1135259372895031297 https://twitter.com/dripstatstatus/status/1135261993055600640

Networking issues between our servers causing downtime for end users

Jul <var data-var='date'>31</var>, <var data-var='time'>03:38</var> UTC Resolved - The metrics have been stable after disabling edge locations, and the problem should not reappear as long as we keep edge locations disabled. We are marking the incident as resolved.Jul <var data-var='date'>31</var>, <var data-var='time'>03:07</var> UTC Monitoring - Through our automated alerting, we were notified of CDN connectivity issues for our primary CDN from South and Southeast Asia, and a corresponding drop in pageviews seen by our backends. The problem appears to have started around 2018-07-31 02:20 UTC, though we have not been able to pinpoint the precise time due to obfuscation of metrics by DNS and content caching. We identified the likely problem as being a misbehaving edge location. To address this, we have temporarily disabled edge locations at around 2018-07-31 02:50 UTC. We reached out to our primary CDN provider for further investigation. After disabling edge locations, our CDN alerts and pageview volume drop alerts have resolved, so that the immediate problem is mitigated. Affected countries include India, Malaysia, Taiwan, and Republic of Korea. Other countries in the South and Southeast Asia (and nearby) were likely also affected. Japan (which had experienced issues in a previous incident, see http://status.liftigniter.com/incidents/5xy6fjyc1rm4) was not affected. The connectivity issues were not experienced by all end users in these regions; in particular, our automated CDN failover health check did not detect a problem with the primary CDN. Due to client-side caching, many users in these regions would have seen no issues.

CDN issues in South and Southeast Asia

Jul <var data-var='date'>21</var>, <var data-var='time'>22:37</var> UTC Resolved - All nodes have been moved to the new, safer, more minimal instance template, and data has been restored. We have also confirmed that the specific issue that caused node crashes is no longer occurring. We have also put together an improved set of best practices around both peacetime capacity changes and emergency responses, so as to minimize data loss. Our cloud provider also created a public issue tracker for their underlying issue at https://issuetracker.google.com/issues/111753610Jul <var data-var='date'>14</var>, <var data-var='time'>01:56</var> UTC Update - We are continuing to monitor for any further issues.Jul <var data-var='date'>14</var>, <var data-var='time'>01:56</var> UTC Monitoring - We have confirmed that there is no data loss, and the services are running fine; however, we lack our usual storage capacity buffer right now. Our cloud provider has confirmed ongoing issues on their side that caused our problems, and an ongoing investigation. In the meantime, they have provided us with guidance on working around the issue to reprovision capacity. We are working on that reprovisioning, and will mark the issue resolved when the reprovisioning is completed.Jul <var data-var='date'>13</var>, <var data-var='time'>21:02</var> UTC Update - The problems turned out to be more serious than expected, with additional nodes affected; we are getting in touch with our cloud provider for more diagnosis and resolution around the issues. For now, the bad nodes have been removed; due to data replication we expect data loss to be minimal and expect to recover data through our standard recovery procedure.Jul <var data-var='date'>13</var>, <var data-var='time'>17:57</var> UTC Identified - Starting 10:20 AM PDT (17:20 UTC) on Friday, July 13, one of our datastore nodes in one region started misbehaving. We received notifications for and began addressing the issue within ten minutes. The node is currently doing a cold restart and we expect it to be back up by 11:45 AM PDT (18:45 UTC). No data has been lost; however, until the node comes fully back up, some data will appear to be missing or unavailable. This has impact on the following services: - Inventory API errors: Customers using the GET, POST, and DELETE endpoints of our inventory API will see error rates and their intended actions may not complete. - Degraded recommendation quality: We'll continue to return recommendations; there is no effect on the overall error rates of our model servers. However, the quality of the recommendations will be degraded and latency will be higher due to the difficulty retrieving all the necessary data to make a great recommendation.

Is LiftIgniter down?

LiftIgniter status is UP

Lastest published events

Backend server overload resulting in issues with recommendation quality as well as update lags

Issues with inventory API and related background processing

Issues with email service (and another component that is not end-user-facing) likely due to Google Cloud infrastructure issues

LiftIgniter Services, Components and Regions

Recent LiftIgniter events

Backend server overload resulting in issues with recommendation quality as well as update lags

Issues with inventory API and related background processing

Issues with email service (and another component that is not end-user-facing) likely due to Google Cloud infrastructure issues

Capacity issues in US East region due to Google Cloud infrastructure issues (they are unable to provision new instances)

Issues with console and the API usage

Issues with services in US East due to capacity issues with cloud provider

Get your FREE account now,
5 monitors included!

Is LiftIgniter down?

LiftIgniter status is UP

Lastest published events

Backend server overload resulting in issues with recommendation quality as well as update lags

Issues with inventory API and related background processing

Issues with email service (and another component that is not end-user-facing) likely due to Google Cloud infrastructure issues

LiftIgniter Services, Components and Regions

Recent LiftIgniter events

Backend server overload resulting in issues with recommendation quality as well as update lags

Issues with inventory API and related background processing

Issues with email service (and another component that is not end-user-facing) likely due to Google Cloud infrastructure issues

Capacity issues in US East region due to Google Cloud infrastructure issues (they are unable to provision new instances)

Issues with console and the API usage

Issues with services in US East due to capacity issues with cloud provider

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!