May <var data-var='date'>20</var>, <var data-var='time'>11:03</var> UTC Completed - The scheduled maintenance has been completed. May <var data-var='date'>20</var>, <var data-var='time'>10:50</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>20</var>, <var data-var='time'>10:46</var> UTC Scheduled - We are performing upgrades on the EU server, NFTX will be unavailble for 60 minutes

Graph Node Upgrade

Sep <var data-var='date'> 2</var>, <var data-var='time'>21:00</var> UTC Resolved - The subgraph has been reindexed. https://thegraph.com/explorer/subgraphs/Epp6gJotEJKAsJYdk7bDEURtGWWeZVXwZds2Pgw8eG51?view=Curators&chain=arbitrum-one Current version v1.4.0 QmYr6m1ESfvhKuZCQxyzy51o7oDWiwGgnHXL9MRZPuVvMr Sep <var data-var='date'> 2</var>, <var data-var='time'>19:24</var> UTC Identified - Two transactions have been reviewed which has uncovered the issue. Transaction #1: https://etherscan.io/tx/0x395710b73d2156c87b11644d10db31b97f0a7a9ffd99c49ed6e988876d9adcab Occurred 5 days ago and was someone buying a PACA NFT from the V2 vaults through Reservoir. Transaction #2: https://etherscan.io/tx/0xa6bbc5d76880b29dae266ce31f9113f36741cb4e9d9888007a466285c1a384bb Occurred a few hours ago and was someone buying a PACA NFT from the V2 vaults through Reservoir. This transaction is the cause of the subgraph outage. The events on both transactions are the same (a few differences in the token holding values of the pool, but that would be expected based on price differences). Looking at the TokenId 559 (https://etherscan.io/token/0x3db5463a9e2d04334192c6f2dd4b72def4751a61?a=559) that was successfully bought in transaction #1, it ended up in the Vault through this transactions https://etherscan.io/tx/0xcd7bb8609e60aa2ac100ead2036eac47ed4982877003d9e4b63785e9ae1cdb2d which was a Safe Transfer (it was a sell into the NFTX vault). Looking at Token ID 9511 (https://etherscan.io/token/0x3db5463a9e2d04334192c6f2dd4b72def4751a61?a=9511) that failed/broke the subgraph, it arrived in the vault through a "Proxy Assert" 1000+ days ago. This is likely due to someone transferring two NFTs directly into the vault instead of going through the mint/sell process, which caused the missing `dateAdded` issue. (https://etherscan.io/tx/0x594a0d7dc671c854084295a0cad461a26e3377cc37a7d1aabde1d00a0e368d52) A fix has been deployed to remove the required value for the mint date, and the subgraph is reindexing now (est 4 hours). Sep <var data-var='date'> 2</var>, <var data-var='time'>19:20</var> UTC Investigating - The NFTX V2 subgraph has failed, we are working on a fix

Subgraph error on null date

Jun <var data-var='date'>28</var>, <var data-var='time'>15:07</var> UTC Resolved - The subgraph has now been upgraded to cover the issues with the fault indexing the transaction. All data is back and running again. Jun <var data-var='date'>28</var>, <var data-var='time'>09:55</var> UTC Identified - A sell evet on NFTX V3 has caused an error on the NFTX V3 vaults subgraph. Any transactions from block https://etherscan.io/block/20183594 hasn't been picked up yet. An update to the subgraph is in progress now, then will be pushed to studio and reindexed. Jun <var data-var='date'>28</var>, <var data-var='time'>08:16</var> UTC Investigating - The indexer that pulls updates from the subgraph is not collecting the latest data. This is causing some buys/sells to not show up against the ongoing activity on V2 and V3 apps. The AMM reporting on NFTX V3 is showing the buys/sells against the pool, however the vaults/collections are not showing the new items in and out of the vault. This may cause some issues when trying to buy and item that no longer belongs in the vault, or selling the item and it not appearing immediately. This will not impact the protocol, all transactions will either fail or succeed, no NFTs or funds are at risk.

Indexer not picking up new vault activity

Jun <var data-var='date'>10</var>, <var data-var='time'>11:25</var> UTC Resolved - The subgraphs have been updated and user holdings are now appearing on the main sell page https://nftx.io/sell/ and individual vault sell pages. Jun <var data-var='date'> 9</var>, <var data-var='time'>08:37</var> UTC Identified - On the sell page on https://nftx.io, as well as the sell pages on the individual vaults, the 721 and 1155 subgraph are not returning NFTs that are held by the connected user. We have identified the issue and are updating the subgraph. In the meantime, you can still sell your items into NFTX through https://explorer.reservoir.tools/ethereum and https://pro.opensea.io/

Owned NFTs showing no holdings

Jun <var data-var='date'> 3</var>, <var data-var='time'>08:44</var> UTC Resolved - The issue was attributed to the Graph Index server. An abundance of requests through the graph led to some queries receiving 400 responses. Additional improvements have been made to the graph server and settings to improve caching and load capabilities. Jun <var data-var='date'> 1</var>, <var data-var='time'>10:48</var> UTC Monitoring - The issue has resolved itself however we're still looking into the root cause. Jun <var data-var='date'> 1</var>, <var data-var='time'>10:15</var> UTC Investigating - There are currently 500 errros on a number of the NFTX V3 API requests. All vaults/funds/nfts are safu.

nftx-v3 API connection issues

Jul <var data-var='date'>10</var>, <var data-var='time'>20:38</var> UTC Resolved - The API has been updated with new keys and is working again. Jul <var data-var='date'>10</var>, <var data-var='time'>19:56</var> UTC Investigating - There is an issue with quote requests hitting the 0x API at the moment. We have identified the issue and are applying a fix.

Buy/Sell/Swap temporarily unavailable

May <var data-var='date'>14</var>, <var data-var='time'>08:34</var> UTC Resolved - The RPC node has been running successfully for the past 24 hours and we are closing this ticket. May <var data-var='date'>13</var>, <var data-var='time'>08:23</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>13</var>, <var data-var='time'>08:23</var> UTC Identified - The RPC servers were returning a 503 response and casuing issues with both the API indexer and the app. These have been fixed and the site is working as expected. May <var data-var='date'>13</var>, <var data-var='time'>07:28</var> UTC Update - We are still seeing buys and sells on the platform, and across other aggregators, however some actions like staking and unstaking are throwing errors on the app. May <var data-var='date'>13</var>, <var data-var='time'>07:26</var> UTC Investigating - The RPC nodes for NFTX are behind the current block and some transactions may fail.

RPC Node outage

Feb <var data-var='date'> 4</var>, <var data-var='time'>19:43</var> UTC Resolved - An update to the Inventory Staking contract has been pushed, which has fixed the temporary bug of new xTokens for existing vaults. Users who staked inventory positions in the past 24 hours will have received the wrong xToken. NFTX will provide a way to migrate these xToken across within the 7 days you will have been locked. Users who had previous staked positions can now unstake if you reach the end of your time lock. Users can now stake their inventory, which will be created on the orginal xToken address. Any fees earned during the 24 hours while the xToken address was changed will have been allocated to the DAO. The DAO will distribute these fees to the staked inventory providers based on a token-holding snapshot for each block where fees were generated. This will be actioned asap within the next two weeks. Any gas costs for the migration from the new xToken back to the old will be reimbursed. A full postmortem of the issue will be released to follow up on this status on https://blog.nftx.io Feb <var data-var='date'> 4</var>, <var data-var='time'>09:21</var> UTC Identified - During a recent update to the Inventory Staking contract, a change in the formatting of the contract caused the byte code hash to change. This change meant that the hash of the byte code was different to the previous version. The hash of this byte code is used to programmatically determine the xToken address when staking NFTs into the pool, and since this was different, any new inventory staking events generated a new xToken address. The app has temporarily disabled inventory staking/unstaking while this is being fixed. Key points - Your NFTs and Staked positions are safe - We have disabled the ability to mint any staked positions on the app that differ from the OG while this is fixed - There have been two versions of some staked positions minted due to a bug, which *does not* impact collateralization nor the ability to redeem - There will be a review of the distribution of inventory fees to ensure that everyone receives their correct amount Feb <var data-var='date'> 4</var>, <var data-var='time'>02:33</var> UTC Update - Any new inventory staking have been created using a new xToken address following an update earlier today. While your positions are still the same as before, we are investigating if rewards will be distributed to these addresses and ensure that the marketplace and yield apps correctly show your positions. The apps are currently only pulling through the existing xToken addresses, so any new staked items in the past 12 hours will not be visible through the app. Feb <var data-var='date'> 4</var>, <var data-var='time'>01:51</var> UTC Investigating - We are experiencing issues with the front-end display of Inventory Staking positions at the moment. Investigations are ongoing, and updates will be provided as more details are discovered.

Inventory Staking/Unstaking Issues

Feb <var data-var='date'> 4</var>, <var data-var='time'>19:27</var> UTC Resolved - We have temporarily disabled the server-side rendering (SSR), which has fixed the issues we were having on the Arbitrum chain. This has a knock-on impact of disabling the social card shares, however it is preferential in this situation while we update the web app to us Rainbow Kit for wallet connections. Feb <var data-var='date'> 4</var>, <var data-var='time'>01:48</var> UTC Identified - There is an issue with our the current implementation of UseDapp that we are using on the Marketplace, which Rainbow kit seems to be working fine. The Marketplace (https://arbitrum.nftx.io) is migrating to rainbow kit to alleviate the issues. Further details will be provided as updates are available Feb <var data-var='date'> 3</var>, <var data-var='time'>11:14</var> UTC Investigating - There is an issue with the network provider on the Arbitrum app at the moment; we are working on finding a solution now.

Arbitrum Network Issues

Jan <var data-var='date'>28</var>, <var data-var='time'>22:48</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>28</var>, <var data-var='time'>22:47</var> UTC Identified - On the CoolCats vault, there was a swap 360 days ago for an item that the user didn't own. This was done directly on Etherscan (otherwise they couldn't do it on the UI), and instead of failing the contract emitted events that made it look like the id was added to the vault and then removed. As a result of this transaction one of the CoolCats was in the vault but not listed as one of the holdings. Earlier today someone swapped the CoolCat out of the vault which caused the subgraph indexing to fail — it was trying to put a date against an item that was removed that the subgraph thought wasn't in the vault, and NULL wasn't an accepted value against the date. To fix the issue we have made the `dateAdded` field on the subgraph an optional field, and the subgraph is working again. The new deployment is `QmeeGQ9xzXRf2vsYwi9MkuU8i4om6R9GUDhdp9q7j4oZyV`, and the decentralised graph and the hosted endpoint have both been updated. Jan <var data-var='date'>28</var>, <var data-var='time'>20:34</var> UTC Investigating - The subgraph providing the data on the site has stopped indexing on block 16506629 after a swap transaction caused an error. We are looking into making an update to this to allow the syncing to resume. This is impacting the updated display for most areas on the site including - current holdings on vaults - new vaults - activity (buy/sell/swap) - staking/unstaking positions Any actions on the site (buying/selling/staking/swapping) will still work, but the results won't appear on the site until the bug is fixed on the subgraph.

Subgraph Outage

Nov <var data-var='date'>21</var>, <var data-var='time'>12:00</var> UTC Completed - The scheduled maintenance has been completed. Nov <var data-var='date'>21</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'>21</var>, <var data-var='time'>09:59</var> UTC Scheduled - The NFTX Graph Node is migrating to new infrastructure. The app may not show the latest data relating to vault holdings, positions, aprs, however the protocol and site will continue to operate as expected.

Graph Node Migration

Nov <var data-var='date'> 9</var>, <var data-var='time'>10:26</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'> 9</var>, <var data-var='time'>10:26</var> UTC Identified - The issue occurred when Hashmasks and Wizards had a free claime which resulted in the cycling of all the NFTs from the vaults. The cycle increased the difference in holdings on the vaults compared with the search holdings by more than 5% which triggered a manual check to ensure this was okay. Now the check is complete and the scripts are running as expected again. Nov <var data-var='date'> 9</var>, <var data-var='time'>09:08</var> UTC Investigating - We have noticed that some of the new items minted into the vaults are not showing up in the search filters. We are revieiwing this issue now and will update once we have identified this problem.

Search Filters Lagging

Nov <var data-var='date'> 1</var>, <var data-var='time'>12:00</var> UTC Completed - The scheduled maintenance has been completed. Nov <var data-var='date'> 1</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>27</var>, <var data-var='time'>12:25</var> UTC Scheduled - We will be undergoing a migration to a new graph node and there may be a short outage on the website during this time.

Oct <var data-var='date'>12</var>, <var data-var='time'>06:43</var> UTC Resolved - The GRT balance has been updated and queries are successfully being served. Oct <var data-var='date'>12</var>, <var data-var='time'>06:43</var> UTC Identified - There was unpaid GRT owing on the API keys that are maintained for Graph requests. Oct <var data-var='date'>12</var>, <var data-var='time'>06:16</var> UTC Investigating - There is an issue with the graph responses on the NFTX marketplace and yield app. These are being looked into now.

Graph responses failing

Sep <var data-var='date'>16</var>, <var data-var='time'>21:31</var> UTC Resolved - With a bleeding-edge patch applied to one of our Erigon nodes the NFTX Graph Indexer was able to resync the NFTX subgraph. The site is now running as expected, and we will continue to closely monitor the service over the weekend. Sep <var data-var='date'>16</var>, <var data-var='time'>11:21</var> UTC Identified - There is an issue with the way in which Erigon is dealing with `trace_block` since the merge. This is resulting in a number of subgraphs that use trace to fail at the same block. We have tried to switch our indexer from Erigon to GETH to fix the issue, however GETH does not support trace so the subgraph still fails. Some updates have been made to the subgraph to remove the reliance on trace and a new subgraph has been deployed to test if this can fix the issue, the sync for this is looking to be 8-12 hours. It is now a race between Erigon pushing a fix for this we can patch our nodes and resume indexing, or once the new subgraph deployment is fully indexed and tested then we can update the endpoints. In the meantime we have reverted requests for the holdings to fallback to Web3 calls. This means that the vaults will load initially with the subgraph data before updating to show only items that are currenlty held in the vaults. NFTX will still not be able to show any recent activity or newly created vaults on the UI. Sep <var data-var='date'>16</var>, <var data-var='time'>09:32</var> UTC Update - This appears to be an issue relating to both The Graph and Erigon. We applied https://github.com/ledgerwatch/erigon/commit/15ad43e1000e29e48f11de26aa42ebde72050772 patch however there were not improvements. There are some indexers which are able to index to the current block now so we are looking at whether those will begin to return the up-to-date data for the apps. Sep <var data-var='date'>15</var>, <var data-var='time'>21:20</var> UTC Investigating - The NFTX subgraph (along with a number of other subgraphs) is failingin on block `15539509`. This is unrelated to the protocol being paused and likely linked to the recent merge. The Graph team is looking into the issue and we're awaiting further updates from them. You will still be able to interact with the app however data related to the holdings, recent transactions, activity etc may be incorrect (i.e. it will be showing the state of the platform as of block `15539509`).

NFTX subgraph syncing issues

Sep <var data-var='date'>15</var>, <var data-var='time'>21:15</var> UTC Resolved - This incident has been resolved. You can read more on https://blog.nftx.io/postmortem-nftxmarketplace0xzap-vulnerability/ Sep <var data-var='date'>15</var>, <var data-var='time'>20:54</var> UTC Monitoring - The fix has been pushed and we're unpausing the contracts now. Sep <var data-var='date'>14</var>, <var data-var='time'>19:57</var> UTC Update - Tests have all passed and the updates to the contact has been staged on Aragon. The next update will be when the votes are enacted in approximately 18 hours. Sep <var data-var='date'>14</var>, <var data-var='time'>09:54</var> UTC Update - The fix has been written and is undergoing testing over the next few hours. Once these tests have passed and the updated contracts reviewed they will be staged on Aragon. Once enacted final tests on mainnet will be conducted and if successful the contract will be unpaused and protocol use will restart. Current estimated time for restarting the protocol is 36 hours from now (2100 UTC Thursday 15th September). Sep <var data-var='date'>13</var>, <var data-var='time'>21:32</var> UTC Identified - We have reviewed the report into the potential vulnerability on one of the periphery contracts and can confirm the vulnerability. Pausing the contracts means that there is no longer risk of exploitation. A fix for the issue is being worked on now and will be tested once completed. Once the tests have passed the updates will be staged and deployed and the protocol will be re-activated. Full postmortem will be annouced after the contracts are re-activated. Sep <var data-var='date'>13</var>, <var data-var='time'>19:18</var> UTC Investigating - NFTX protocol is paused while we investigate a report into a potential vulnerability on a periphery contract.

NFTX protocol paused

Sep <var data-var='date'> 1</var>, <var data-var='time'>19:28</var> UTC Resolved - The NGINX Proxy servers were getting errors on the response causing the response causing issues. The fix has been applied and the metadata is back. Sep <var data-var='date'> 1</var>, <var data-var='time'>17:20</var> UTC Investigating - The metadata API is currently experiencing some issues and images/metadata may not be loading. We are looking into the issue now.

Metadata API Issues

Aug <var data-var='date'>18</var>, <var data-var='time'>22:33</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>18</var>, <var data-var='time'>14:07</var> UTC Monitoring - The issue recetified itself, we are looking into what caused it. Aug <var data-var='date'>18</var>, <var data-var='time'>14:04</var> UTC Investigating - There is currently issues with the NFTX Subgraph endpoint, we are looking into this now.

Graph endpoint errors

Jul <var data-var='date'>19</var>, <var data-var='time'>08:22</var> UTC Resolved - The search index has now completed and filters are back active on all vaults. To combat these recent issues we are now taking nightly backups of the search index to reduce reliance on the Algolia infrastructure. Jul <var data-var='date'>18</var>, <var data-var='time'>12:40</var> UTC Identified - The filters are currently unavailable on the vault pages for all mainnet collections. We have identified the issue with the search index and are reindexing now. These will be back in the next few hours.

NFT Search Filters Unavailable

Jul <var data-var='date'>17</var>, <var data-var='time'>13:41</var> UTC Resolved - This was an issue on the decentralised network and is back working again. https://status.thegraph.com/incidents/4zn6n5dtzbhd Jul <var data-var='date'>17</var>, <var data-var='time'>11:20</var> UTC Investigating - We are currently seeing issues with the subraph endpoint and are working on a new deployment to fix these. Vaults may not appear with the latest details/holdings.

Subgraph Endpoint Issues

Jun <var data-var='date'>21</var>, <var data-var='time'>07:11</var> UTC Resolved - The issue has been resolved by Cloudflare. You can find out more about the issue by reviewing https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj Jun <var data-var='date'>21</var>, <var data-var='time'>06:45</var> UTC Identified - Cloudflare is currently experiencing outages which will be impacting most of the internet this morning. The NFTX endpoings utilise Cloudflare which is causing the vaults/collections etc to not load. Once Cloudflare have corrected their issue the site will be back available.

Cloudflare outage impacting endpoints

Jun <var data-var='date'>16</var>, <var data-var='time'>16:21</var> UTC Resolved - The index has completed and all filters are working again as expected. Jun <var data-var='date'>16</var>, <var data-var='time'>07:14</var> UTC Identified - The records have dropped from our Algolia search provider. We are looking to restore the index from yesterdays backup while also rebuilding the index from scratch (this takes a few hours). We have also indentified that an error with the graph response caused the issue and will patch the script to avoid any false responses causing these issues. Jun <var data-var='date'>16</var>, <var data-var='time'>07:12</var> UTC Investigating - The search filters on our collections are currently returning no options/results.

Search Filters

Mar <var data-var='date'>31</var>, <var data-var='time'>23:30</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>31</var>, <var data-var='time'>23:30</var> UTC Identified - The new subgraph was deployed but the indexing wasn't up-to-date. This was fixed and re-indexed, everything working as expected again now. Mar <var data-var='date'>31</var>, <var data-var='time'>22:57</var> UTC Investigating - Subgraph has stopped responding and we're not getting vaults and holding data back. We are looking into this now.

SubGraph Incident

Jan <var data-var='date'>21</var>, <var data-var='time'>12:48</var> UTC Resolved - The API issue seems to be resolved now. We are looking into making Covalent our preferred metadata API provider while potentially building our own solution that indexes directly off the contracts. Jan <var data-var='date'>20</var>, <var data-var='time'>23:57</var> UTC Identified - OpenSea have experienced high levels of traffic and have disabled a number of API users. We have reached out to get our keys added back to the allow list and are still awaiting feedback from Opensea. Jan <var data-var='date'>20</var>, <var data-var='time'>15:24</var> UTC Investigating - OpenSea are currently experiencing an outage on their site and API. The API issues are impacting the display of the images and some metadata around the items within the vaults. Where we already have cached the assets you can see them, but any not cached will not load at the moment. https://status.opensea.io/incidents/88862t31ybw5

Some NFT image/details not appearing

Jan <var data-var='date'>20</var>, <var data-var='time'>23:56</var> UTC Resolved - While we were waiting for the shared hosting on The Graph to index we had a temporary infrasturcture which couldn't handle the additional traffic after the relaunch. This has been replaced with the graph index node that will be used for the decentralised graph index moving fowards. This has been operational for a few hours now with no further reported issues. Jan <var data-var='date'>20</var>, <var data-var='time'>17:46</var> UTC Investigating - Some of the staking pages are not reflecting your permissions due to an issue with the requests to our subgraph. We are looking into the issue now.

Is NFTX down?

NFTX status is UP

Lastest published events

Graph Node Upgrade

Subgraph error on null date

Indexer not picking up new vault activity

NFTX Services, Components and Regions

Recent NFTX events

Graph Node Upgrade

Subgraph error on null date

Indexer not picking up new vault activity

Owned NFTs showing no holdings

nftx-v3 API connection issues

Buy/Sell/Swap temporarily unavailable