Apr <var data-var='date'> 6</var>, <var data-var='time'>11:32</var> PDT Resolved - Following an extended period of stability, we have verified that the tactical fix applied to our database environment has fully resolved the connectivity issues and all affected services are performing at baseline levels. We appreciate your patience throughout this event.Apr <var data-var='date'> 2</var>, <var data-var='time'>20:10</var> PDT Monitoring - Following our investigation with the database vendor, a tactical fix was successfully applied at 16:26 PST. Since the implementation of this fix, we have observed a significant improvement in database connection stability and a return to normal service levels. Our engineering team is continuously monitoring the environment to verify the long-term effectiveness of the resolution and ensure overall system stability. We will provide a final update once we have confirmed the issue is fully resolved and appreciate your continued patience.Apr <var data-var='date'> 2</var>, <var data-var='time'>13:31</var> PDT Identified - We have observed a database connectivity issue that intermittently impacted Marqeta services on April 1st between 16:04 and 17:56 PST. During this window, some customers may have experienced an elevated amount of STIPs as well as partial disruptions to tokenization and API traffic. Our engineering team has traced the instability to an intermittent database connection failure and has escalated the matter to our third-party vendor for support. We are currently engaged in an active troubleshooting with the vendor's support to isolate the root cause and prevent recurrence by applying a tactical solution. We apologize for any inconvenience this incident may caused, and will periodically provide updates once available.

Intermittent STIPs across multiple networks

Apr <var data-var='date'> 3</var>, <var data-var='time'>21:15</var> PDT Resolved - Engineers have resolved the last known instances of this issue.Apr <var data-var='date'> 3</var>, <var data-var='time'>19:55</var> PDT Update - Engineers are continuing to work to resolve the last few instances of this issue. We will update here once this work is complete.Apr <var data-var='date'> 3</var>, <var data-var='time'>12:44</var> PDT Identified - From March 24th, we are experiencing an increase in the rate of declines for card-present transactions using magnetic stripe swipe (magstripe). This may cause cardholder's physical cards to be declined even though the cards and accounts are in good standing. Engineers are working to resolve this issue today. We will update here once this work is complete.

Increase Rate Of Magstripe Declines

Mar <var data-var='date'>17</var>, <var data-var='time'>12:27</var> PDT Resolved - All bank files have been successfully delivered as of 12:17 PM PT. No further delays are expected.Mar <var data-var='date'>17</var>, <var data-var='time'>09:09</var> PDT Identified - We are currently experiencing delays in bank file delivery for some customers. Engineers have identified the issue and restarted the delivery process. All bank files are expected to be delivered within the next 3 hours.

Delay in bank file delivery

Mar <var data-var='date'>12</var>, <var data-var='time'>17:50</var> PDT Resolved - We are confirming that the Mastercard network disturbance, which caused two isolated periods of elevated (STIP) earlier today around 11:14 AM and 12:45 PM Pacific Time, is now fully resolved and the service is operating normally. Our engineering team monitored the service to ensure stability with no abnormality observed.Mar <var data-var='date'>12</var>, <var data-var='time'>13:29</var> PDT Monitoring - Marqeta Engineering team observed network disturbance resulting in STIPS on the Mastercard network, We observed two distinct periods of impact, starting around 11:14 AM and 12:45 PM Pacific Time , the status is currently stable and our team is actively engaged with MasterCard to understand the root cause. We are continuously monitoring the network to ensure stability.

STIPS on Master Card network

Mar <var data-var='date'> 4</var>, <var data-var='time'>09:05</var> PST Resolved - Processing has returned to its normal schedule. We are continuing to monitor the Federal Reserve Bank ACH Network for any reoccurrence and will provide updates if any new delays are identified.Mar <var data-var='date'> 3</var>, <var data-var='time'>18:04</var> PST Update - We have processed all ACH transactions received as of this time. However, according to the Federal Reserve Bank's Service Status, there may be continued delays affecting the next set of incoming transactions to the bank. We are continuing to monitor the ongoing issue at the Federal Reserve Bank ACH Network and will provide updates if any further delays are identified.Mar <var data-var='date'> 3</var>, <var data-var='time'>17:22</var> PST Update - We have processed all ACH transactions received as of this time. However, according to the Federal Reserve Bank's Service Status, there may be continued delays affecting the next set of incoming transactions to the bank. We are continuing to monitor the ongoing issue at the Federal Reserve Bank ACH Network and will provide updates if any further delays are identified.Mar <var data-var='date'> 3</var>, <var data-var='time'>13:32</var> PST Update - We are continuing to monitor the ongoing issue at Federal Reserve Bank ACH Network which may lead to a delay in the processing of ACH transactions. We have not yet been provided an ETA for resolution or any workarounds. We will update here as soon as they indicate that the issue has been resolved and is stable.Mar <var data-var='date'> 3</var>, <var data-var='time'>08:16</var> PST Investigating - Marqeta has been notified of an ongoing issue at Federal Reserve Bank ACH Network which may lead to a delay in the processing of ACH transactions. This can affect Direct Deposits, and ACH Origination. We have not yet been provided an ETA for resolution of their issue. We will update you as soon as they indicate that the issue has been resolved and is stable.

FEDACH Processing Delay

Feb <var data-var='date'>25</var>, <var data-var='time'>10:07</var> PST Resolved - At approximately 9:23 AM PT, Marqeta customers may have experienced issues logging into Marqeta Dashboard. Engineers quickly identified the root cause and implemented a fix. Customers can now access the Dashboard without issues.

Marqeta Dashboard Login Outage

Feb <var data-var='date'>13</var>, <var data-var='time'>14:25</var> PST Resolved - The full fix has now been successfully implemented across all Pulse routers as of February 13th approximately 2:15 PM PT. This follows the previously completed Visa rollback on February 10th, and the fix is now fully generalized across all networks. The implementation was executed in coordination with our network partners and completed as planned. We continue to closely monitor the environment and confirm that no duplicate transactions have been observed since Friday, February 6th at 6:00 PM PT / 2:00 AM UTC. The system remains stable, and no additional impact is anticipated to card networks. We appreciate your patience throughout this process.Feb <var data-var='date'>11</var>, <var data-var='time'>09:34</var> PST Update - The full rollback for Visa was successfully completed in close coordination with the Visa team on February 10th at 9:30 PM PT, The implementation was executed smoothly, and no abnormality was observed. For Pulse, we are actively coordinating next steps with their team. For an aligned maintenance window for fix implementation to ensure proper timing and minimize any potential downtime. We continue to closely monitor the environment and confirm that no duplicate transactions have been observed since Friday, February 6th at 6:00 PM PT / 2:00 AM UTC. At this time, we do not anticipate additional impact to either the Visa or Pulse card networks. We will provide further updates as additional information becomes available and appreciate your continued patience.Feb <var data-var='date'>10</var>, <var data-var='time'>12:32</var> PST Update - A non-impact partial fix for the issue was applied to Pulse on February 9th at 06:30 PM PT. Our engineering team is currently working with priority on finalizing the full-scale fix, and we will provide an update as soon as it is ready for deployment. We continue to confirm that no duplicate transactions have been observed since Friday, February 6th at 6:00 PM PT/2:00 AM UTC. We do not expect any further impact to either the Visa or Pulse card networks. We deeply appreciate your patience and remain fully committed to providing explicit communication throughout the resolution process.Feb <var data-var='date'> 9</var>, <var data-var='time'>13:41</var> PST Update - Engineers are scheduling the fix for Visa and Pulse. No duplicate transactions have been observed since Friday, February 6th at 6:00 PM PT/2:00 AM UTC. We do not expect any further impact to either card network.Feb <var data-var='date'> 6</var>, <var data-var='time'>18:05</var> PST Update - Our team has identified the cause of duplicate transaction postings as an issue introduced in a recent release. We've successfully completed a rollback for Mastercard. As a precautionary measure, we're scheduling the Visa and Pulse fixes for Monday, February 9th to avoid any potential impact during the weekend. All duplicate transactions will be automatically reversed within 48 hours, and no action is required from affected customers. We will provide our next update on Monday unless new information becomes available sooner.Feb <var data-var='date'> 6</var>, <var data-var='time'>16:00</var> PST Identified - We've identified the cause of the duplicate transaction issue and are implementing a fix.Feb <var data-var='date'> 6</var>, <var data-var='time'>15:16</var> PST Update - Our team is conducting a thorough investigation to determine the root cause while working on a resolution. We will provide updates as soon as new information becomes available.Feb <var data-var='date'> 6</var>, <var data-var='time'>12:45</var> PST Investigating - Starting January 22nd, some customers may notice duplicate transaction postings. Our engineering team is actively working to resolve this issue. Affected customers do not need to take any action.

Duplicate transaction postings

Feb <var data-var='date'>13</var>, <var data-var='time'>06:35</var> PST Resolved - The issue causing elevated API timeouts between 2:00 PM – 3:22 PM PT (22:00 – 23:22 UTC) on Thursday, February 12th has been fully resolved. The root cause was identified and mitigated, and our engineering team has continued to monitor the environment closely. No further abnormalities have been observed. We do not anticipate any additional impact at this time. Thank you for your patience.Feb <var data-var='date'>12</var>, <var data-var='time'>17:15</var> PST Monitoring - Some Marqeta customers may have experienced elevated API timeouts between 2:00 PM - 3:22 PM PT (22:00 - 23:22 UTC) on Thursday, February 12th. The root cause has been identified, and we do not anticipate any further impact at this time.

Elevated API Timeouts

Feb <var data-var='date'>11</var>, <var data-var='time'>06:21</var> PST Resolved - This has now been resolved.Feb <var data-var='date'>11</var>, <var data-var='time'>01:51</var> PST Investigating - Affected customers using the 2.0 simulation suite will be unable to use the simulations/cardtransactions/ and simulations/directdeposits/ apis, receiving a 503 response. We are escalating to the Product owners, we will update when they have identified the issue.

Sandbox simulations/cardtransactions/ endpoint is reporting 503

Feb <var data-var='date'> 2</var>, <var data-var='time'>11:30</var> PST Resolved - Engineers have confirmed that the increase in STIPs occurred between 10:29 AM PT - 10:35 AM PT due to a networking issue in one region. It is now resolved.Feb <var data-var='date'> 2</var>, <var data-var='time'>11:06</var> PST Monitoring - Engineers have diverted traffic from the impacted area. Customers should no longer be impacted. We will continue to monitor the situation and provide updates when new information becomes available.Feb <var data-var='date'> 2</var>, <var data-var='time'>11:01</var> PST Investigating - At approximately 10:29 AM PT Marqeta engineers observed an elevated amount of STIPs. We are currently investigating the issue. The next update will be at 11:30 AM PT.

STIPs across all networks

Jan <var data-var='date'>27</var>, <var data-var='time'>10:06</var> PST Resolved - Engineers have confirmed the reverted deployment has restored functionality. Teams will continue working to understand the root cause of the degraded functionality.Jan <var data-var='date'>27</var>, <var data-var='time'>09:51</var> PST Investigating - Marqeta engineers have been informed that Tokenmanager in the EU is experiencing an outage. Engineers are reverting a deployment that should restore full functionality shortly.

Tokenmanager EU is Down

Jan <var data-var='date'>24</var>, <var data-var='time'>07:20</var> PST Resolved - As of 7:15 AM Pacific time all Clearing Files have been processed and delivered.Jan <var data-var='date'>23</var>, <var data-var='time'>23:43</var> PST Monitoring - Clearing processing is operating as expected as of 11:40 PM Pacific time. The earlier errored condition has created a backlog which is being processed at this time. Clearing files will be delivered as they are created and we expect all files to be delivered by 8 AM Pacific time Saturday, January 24th.Jan <var data-var='date'>23</var>, <var data-var='time'>23:12</var> PST Identified - At 3 PM Pacific time on January 23rd the Marqeta clearing process experienced an errored condition. This error has been resolved and the clearing process has begun. Clearing files for our customers will be delayed up to 8 hours as a result of the error.

Clearing files are delayed

Jan <var data-var='date'>22</var>, <var data-var='time'>15:12</var> PST Resolved - Marqeta experienced an outage today affecting multiple services. Once the team isolated the root cause to a networking configuration issue, a fix was deployed and all impacted services were fully restored. The recovery is holding across all services. If you have any questions related to this matter, please reach out to support@marqeta.com.Jan <var data-var='date'>22</var>, <var data-var='time'>12:27</var> PST Monitoring - The team has isolated root cause to a networking configuration issue. A fix has been deployed and services have recovered. The teams are continuing to isolate and address any lingering effects of the outage. Further updates will be provided here as they are determined.Jan <var data-var='date'>22</var>, <var data-var='time'>12:15</var> PST Update - Marqeta is seeing multiple services impacted. This will include 500-level API errors, Tokenization Failures, Payment Processing Delays, Dashboard and Reporting Issues, KYC and Customer Onboarding, and Webhooks Functionality. The team has isolated root cause to a networking configuration issue. A fix has been deployed and most services have recovered. The teams are isolating and addressing any subsequent effects of the outage. We will be able to provide an ETA for full restoration once we are confident the recovery is holding across all services and all subsequent effects are known and addressed. We will update next as soon as restoration is confirmed.Jan <var data-var='date'>22</var>, <var data-var='time'>11:45</var> PST Update - Marqeta is seeing multiple services impacted. This will include 500-level API errors, Tokenization Failures, Payment Processing Delays, Dashboard and Reporting Issues, KYC and Customer Onboarding, and Webhooks Functionality. The team has isolated root cause to a networking configuration issue. A fix has been deployed and we are seeing services recovering. The teams are isolating and addressing any subsequent effects of the outage. We will be able to provide an ETA for full restoration once we are confident the recovery is holding across all services. We will update next at 12:15 PST or as soon as restoration is confirmed.Jan <var data-var='date'>22</var>, <var data-var='time'>11:30</var> PST Update - Marqeta is seeing multiple services impacted. This will include 500-level API errors, Tokenization Failures, Payment Processing Delays, Dashboard and Reporting Issues, KYC and Customer Onboarding, and Webhooks Functionality. The team has isolated root cause to a networking configuration issue. A fix has been deployed and we are seeing services recovering. We will be able to provide an ETA for full restoration once we are confident the recovery is holding across all services. We will update next at 11:45 PST or as soon as restoration is confirmed.Jan <var data-var='date'>22</var>, <var data-var='time'>11:15</var> PST Identified - Marqeta is seeing multiple services impacted. This will include 500-level API errors, Tokenization Failures, Payment Processing Delays, Dashboard and Reporting Issues, KYC and Customer Onboarding, and Webhooks Functionality. We have multiple teams engaged and triaging. The team has isolated root cause to a networking configuration issue. A fix has been deployed and we are seeing signs of recovery. We will be able to provide an ETA for full restoration once we are confident the recovery is holding. We will update next at 11:30 PST or as soon as restoration is confirmed.Jan <var data-var='date'>22</var>, <var data-var='time'>11:00</var> PST Update - Marqeta is seeing multiple services impacted. We have multiple teams engaged and triaging. The team has isolated root cause to a networking configuration issue. We are evaluating resolution paths presently, but do not yet have an ETA for restoration. We will update next at 11:15 PST.Jan <var data-var='date'>22</var>, <var data-var='time'>10:45</var> PST Update - Marqeta is seeing multiple services impacted. We have multiple teams engaged and triaging. The team has isolated root cause to a networking configuration issue. We are evaluating resolution paths presently, but do not yet have an ETA for restoration. We will update next at 11:00 PST.Jan <var data-var='date'>22</var>, <var data-var='time'>10:30</var> PST Update - Marqeta is seeing multiple services impacted. We have multiple teams engaged and triaging. The team has isolated root cause to a networking configuration issue. We will update next at 10:45 PST.Jan <var data-var='date'>22</var>, <var data-var='time'>10:14</var> PST Update - Marqeta is seeing multiple services impacted. We have multiple teams engaged and triaging. We do not yet have root cause. We will update next at 10:30 PST.Jan <var data-var='date'>22</var>, <var data-var='time'>09:56</var> PST Investigating - We have identified multiple outages across multiple services. Marqeta engineers are working hard to assess the scope of the outage, determine root cause, and provide timelines for restoration. Next update will be at 10:15 PST

Service Impact To Multiple Services

Jan <var data-var='date'>19</var>, <var data-var='time'>09:59</var> PST Resolved - We have implemented mitigation steps and see no further recurrence of this incident since those changes were implemented.Jan <var data-var='date'>19</var>, <var data-var='time'>05:19</var> PST Monitoring - The large spike in abnormal traffic has abated. We are still working to understand why this merchant sent so much traffic to us.Jan <var data-var='date'>19</var>, <var data-var='time'>04:50</var> PST Identified - Cardholders may have experienced elevated authentication declines, increased processing latency, and failed 3DS card authentications since 12:04 PM GMT/7:04 AM EST. The issue is caused by a large amount of abnormal traffic generated by a bad merchant. Our engineering teams identified and isolated the traffic source, and are working towards potential improvements.

3DS authentications impacted, 19th January

Jan <var data-var='date'> 8</var>, <var data-var='time'>12:22</var> PST Completed - The scheduled upgrade to our public sandbox databases has been successfully completed. Post-maintenance verifications have been performed, and system functionality has been confirmed. Thank you for your patience during this maintenance window.Jan <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Jan <var data-var='date'> 8</var>, <var data-var='time'>07:39</var> PST Update - We will be undergoing scheduled maintenance during this time.Jan <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> PST Update - We will be undergoing scheduled maintenance during this time.Jan <var data-var='date'> 8</var>, <var data-var='time'>11:00</var> PST Scheduled - Marqeta will be performing a scheduled upgrade to our public sandbox databases to enhance overall reliability and security. During the maintenance window, public sandbox environments may experience intermittent unavailability. The expected duration of this maintenance is approximately one hour. We appreciate your patience as we complete this work and will provide updates once available

Public Sandbox Database Upgrade

Jan <var data-var='date'> 7</var>, <var data-var='time'>02:01</var> PST Resolved - All clearings have been posted from the affected file. If you notice any discrepancies, please raised them with Marqeta support.Jan <var data-var='date'> 6</var>, <var data-var='time'>15:45</var> PST Update - The corrupted Visa file has been repaired and is now processing. Reporting should be completed shortly. We do not yet have an ETA for when transactions will be cleared, one will be shared here once it has been determined.Jan <var data-var='date'> 6</var>, <var data-var='time'>11:45</var> PST Update - Marqeta engineers have determined that Visa's corrupted file may not be possible to repair. We are continuing to wait for Visa's Clearing team to provide updates. We will update here once Visa sends a corrected file. We have no ETA as yet from Visa for this.Jan <var data-var='date'> 6</var>, <var data-var='time'>09:44</var> PST Update - We are continuing to wait for Visa's Clearing team to provide updates. In the meantime, Marqeta engineers are investigating the possibility of repairing Visa's corrupted file. We will update here once Visa sends a corrected file. We have no ETA as yet from Visa for this. We will also update here if the file repair approach is possible.Jan <var data-var='date'> 6</var>, <var data-var='time'>06:20</var> PST Update - Visa has escalated the issue to their Clearing team for investigation. Once their investigation in complete they will be able to provide us with an corrected file. Affected clearings will be processed once received. We will update here once Visa send a corrected file. We have no ETA as yet from Visa for this.Jan <var data-var='date'> 6</var>, <var data-var='time'>02:46</var> PST Identified - Some clearing transactions are delayed due to a corrupted file from Visa containing malformed characters. We've notified Visa and are waiting for a corrected file. Affected clearings will be processed once received. We will update here once Visa send a corrected file. We have no ETA as yet from Visa for this.

Delayed Visa Clearings January 6th

Dec <var data-var='date'>18</var>, <var data-var='time'>16:51</var> PST Resolved - As of 4:40 PM PT, engineers confirmed that all files have completed processing. Clearing webhooks and reporting should be functioning as expected.Dec <var data-var='date'>18</var>, <var data-var='time'>08:57</var> PST Update - For Mastercard, program level split files are continuing to progress with no ETA yet. Clearing webhooks have an ETA of one hour. Engineers are investigating reports of missing Mastercard T112 reports, and will look to resend them by end of day. While we don't have an exact ETA for complete resolution at this time, we will continue to provide additional details as they become available.Dec <var data-var='date'>17</var>, <var data-var='time'>17:36</var> PST Update - Visa files have finished processing. Engineers have completed the program level split files and they are now currently processing; we will continue to monitor their progress. While we don't have an exact ETA for complete resolution at this time, we will provide additional details as they become available. The next update will occur when all files have been fully processed and we can confirm operational status.Dec <var data-var='date'>17</var>, <var data-var='time'>08:20</var> PST Update - Engineers have clarified that Mastercard files were also impacted in the same timeframe. We are continuing to monitor the processing of Visa and Mastercard clearing files. Currently, the engineers have re-created the program level split files for clearing and reporting. As a part of this work there is a focus on preventing duplicate files from being sent. Once complete, we will send the split files to be cleared and/or delivered to customers. The next update will occur when we have confirmed that all files are caught up.Dec <var data-var='date'>16</var>, <var data-var='time'>03:17</var> PST Update - Engineers found the root cause, and have mitigated. The files are now being processed, Marqeta customers will start to see clearing webhooks now. We will monitor the procesing of clearing files, and update this incident when we are caught up, and will note that we are Operational then.Dec <var data-var='date'>16</var>, <var data-var='time'>02:51</var> PST Investigating - We are investigating issues processing Visa clearing and settlement files for today. We see no issues with clearings and settlement files for other networks. We see no impact to authorizations from any Network. Since the files have not been processed, impacted Marqeta customers on Visa will see an absence of clearing webhooks, and no Visa VSS report files until this incident is resolved. Prrefunded customers will see no clearings posted to cardholder balances until this incident is mitigated.

Clearings on Visa and Mastercard Delayed, 16 December

Nov <var data-var='date'>14</var>, <var data-var='time'>08:13</var> PST Resolved - Visa and Pulse reports should now be caught up for all customers. Visa clearing should now be caught up for all customers.Nov <var data-var='date'>13</var>, <var data-var='time'>10:50</var> PST Update - Pulse reports from today should be delivered for all customers. The Visa clearing is continuing to process. We now expect this to be caught up by tomorrow morning.Nov <var data-var='date'>13</var>, <var data-var='time'>09:49</var> PST Update - Visa reports are processed and delivery should be caught up for all customers. For Pulse the reports from before November 13th have been delivered. Pulse reports from today are still processing. These files will be processed and delivered today. For clearing on Visa, we are still working through the backlog of the files before November 13th. We expect this to be caught up today.Nov <var data-var='date'>13</var>, <var data-var='time'>08:42</var> PST Update - Engineers have resolved the root cause affecting the processing of Visa and Pulse clearing and reports. Visa reports are processed and delivery should be caught up for all customers. For Pulse the reports from before November 13th have been delivered. Pulse reports from today are still processing. These files will be processed and delivered today. For clearing on Visa, we are still working through the backlog of the files before November 13th. We expect this to be caught up today.Nov <var data-var='date'>13</var>, <var data-var='time'>07:22</var> PST Update - We are continuing to work on a fix for this issue. Full resolution is expected today.Nov <var data-var='date'>12</var>, <var data-var='time'>20:11</var> PST Update - We are continuing to work on a fix for this issue. Full resolution is expected tomorrow morning.Nov <var data-var='date'>12</var>, <var data-var='time'>17:12</var> PST Update - Engineering teams are actively working to restore file processing. We will continue to share updates and new information as it becomes available.Nov <var data-var='date'>12</var>, <var data-var='time'>14:39</var> PST Update - Engineering teams continue efforts to restore file processing capabilities. Additional updates and timelines to be communicated as they become available.Nov <var data-var='date'>12</var>, <var data-var='time'>12:36</var> PST Update - Engineers are continuing to work to restore file processing. They anticipate this will be resolved today. Additional updates and time frames will be provided here once they are known.Nov <var data-var='date'>12</var>, <var data-var='time'>11:32</var> PST Update - Engineers have confirmed that Pulse files are also affected. Engineers are continuing to actively investigate to mitigate the impact and resolve the issue.Nov <var data-var='date'>12</var>, <var data-var='time'>10:03</var> PST Identified - Issue was identified to be an internal service traffic issue that is impacting some of the file processing on Visa network, and Engineering team is currently working on developing a fix to address the reporting and clearing services issue. In the mean time, we are processing existing visa network files at reduced capacity.Nov <var data-var='date'>12</var>, <var data-var='time'>07:35</var> PST Investigating - On Nov 11, Marqeta observed a technical issue that is disturbing the communication of some of the clearing & settlement files on Visa Network. Our teams are actively investigating to mitigate the impact and resolve the issue.

Marqeta customers might observe a delay in settlement and clearing processes in Visa Network

Nov <var data-var='date'>13</var>, <var data-var='time'>17:16</var> PST Resolved - After close monitoring, engineers have confirmed that the issue is now stable. Services should be functioning as expected.Nov <var data-var='date'>13</var>, <var data-var='time'>14:25</var> PST Monitoring - At 13:20 PT, Marqeta's team deployed a hot fix to mitigate the impact and solve the issue with Dispute services ,and post deployment service verification showed steady gradual rebound along with multiple service verifications with our customers. Engineering teams are still engaged for service close monitoring tell end of business day.Nov <var data-var='date'>13</var>, <var data-var='time'>13:24</var> PST Update - Marqeta Engineers are sill developing a fix to mitigate the impact on our customers with priority, and service status updates will be shared once availableNov <var data-var='date'>13</var>, <var data-var='time'>12:19</var> PST Identified - Marqeta engineers are preparing a temporary hot fix to mitigate the impact on our customers, and investigations are ongoing to isolate the issue permanently.Nov <var data-var='date'>13</var>, <var data-var='time'>11:55</var> PST Investigating - Marqeta has identified an issue that is impacting Risk & Dispute services we provide. We are actively working to resolve the issue as quickly as possible

Impact on Risk & Disputes services

Nov <var data-var='date'> 7</var>, <var data-var='time'>16:32</var> PST Resolved - We have confirmed that all files have been processed and delivered.Nov <var data-var='date'> 7</var>, <var data-var='time'>14:59</var> PST Identified - There has been a delay in the delivery of Visa SMS and VSS Detail files today. The delay was caused by a recent system change that is preventing the reports from being delivered to the designated SFTP location. We are rolling back the change to remediate the situation.

Delay in Delivering VISA Detail and SMS Files

Oct <var data-var='date'>30</var>, <var data-var='time'>12:40</var> PDT Resolved - The Visa transaction processing incident has been fully resolved. All STIP backlogs have been processed and engineers have completed the deployment of fixes to address the root cause of processing issue.Oct <var data-var='date'>22</var>, <var data-var='time'>10:05</var> PDT Update - The backlog of STIPs has been processed. Engineers are still working on an update to address the root cause of the STIPs.Oct <var data-var='date'>21</var>, <var data-var='time'>20:34</var> PDT Update - The deployment to all Visa routers is complete and we are seeing a steady decline in the backlog of STIPs. This process should be complete by about midnight PST.Oct <var data-var='date'>21</var>, <var data-var='time'>17:26</var> PDT Update - Engineers are remediating a delay in STIP processing for Visa customers. STIPs will become unblocked and customers will start to receive them within the next few hours.Oct <var data-var='date'>21</var>, <var data-var='time'>12:54</var> PDT Update - Ongoing monitoring has observed STIPs still occurring for some customers. Engineers are actively working on a resolution.Oct <var data-var='date'>20</var>, <var data-var='time'>23:43</var> PDT Monitoring - Transactions have continued to process at an expected rate. We will continue to monitor through the night.Oct <var data-var='date'>20</var>, <var data-var='time'>23:21</var> PDT Update - Visa has reverted the change that led to the impairment and we are seeing transactions return to normal levels. Marqeta customers will recieve an elevated number of STIPs as transactions that were attempted to be processed during the event will now be reported.Oct <var data-var='date'>20</var>, <var data-var='time'>22:48</var> PDT Update - We’re working with Visa to address an issue where VisaNet traffic in North America is currently not reaching Marqeta. This results in authorization requests not processing through Marqeta and increased network stand-in (STIP) activity. Customers will see intermittent timeouts/declines and elevated stand-in (STIP) rates. Current impact Scope: North America VisaNet authorizations Effect: Significant drop in authorization traffic; increased stand-in (STIP) Data: No indication of data loss or security impact What we’re doing Working with Visa to stabilize station availability and validate expected behavior. Customer action No action required. If you see persistent anomalies, please contact Marqeta Production Support at support@marqeta.com with example transaction IDs and timestamps. We are actively monitoring this incident. Next update We will continue to post updates here as we progress toward full remediation.Oct <var data-var='date'>20</var>, <var data-var='time'>21:57</var> PDT Update - We are in a working bridge with Visa to resolve the drop in transactions.Oct <var data-var='date'>20</var>, <var data-var='time'>21:37</var> PDT Update - Visa implemented a change and it resulted in a temporary drop in Visa transactions. A response team is engaged with Visa and working to restore transactions.Oct <var data-var='date'>20</var>, <var data-var='time'>18:57</var> PDT Update - Engineers are currently developing a fix for the identified issue , further updates will be shared once fix is deployed and service performance verified.Oct <var data-var='date'>20</var>, <var data-var='time'>16:11</var> PDT Update - After further investigation, Marqeta engineers have isolated the issue to a release-related change on Visa's side. An incident has been escalated to Visa for resolution. We are seeing a decrease in the amount of STIPS. Our engineers are actively working with Visa to resolve this issue and determine the next steps for full restoration.Oct <var data-var='date'>20</var>, <var data-var='time'>07:57</var> PDT Identified - Marqeta Engineers isolated the issue to be related to the ongoing partial degradation to KYC Services resulted from AWS outageOct <var data-var='date'>20</var>, <var data-var='time'>07:12</var> PDT Investigating - Marqeta Engineers were alerted to an elevated amount of STIPs from the Visa Network observed at 06:30 PT. We have a response team engaged and we are investigating.

STIPs on the Visa Network

Oct <var data-var='date'>28</var>, <var data-var='time'>12:31</var> PDT Resolved - Engineers have confirmed that the program selector should be working as intended.Oct <var data-var='date'>28</var>, <var data-var='time'>09:16</var> PDT Investigating - Some customers may find some features of Marqeta Dashboard report customization are unavailable. This will include program filtering. Engineers are aware and are working on restoring all features.

Marqeta Dashboard Report Customization Partial Outage

Oct <var data-var='date'>22</var>, <var data-var='time'>06:58</var> PDT Resolved - Bank files and MQD / DIVA API reports are now refreshed and up to date for the October 20th data.Oct <var data-var='date'>22</var>, <var data-var='time'>06:30</var> PDT Monitoring - MQD / DIVA API reports are now refreshed and up to date for the October 20th data.Oct <var data-var='date'>21</var>, <var data-var='time'>05:40</var> PDT Identified - Bank files data for October 20th is incomplete as a consequence of AWS outage, causing a delay for bank files for that day. MQD/DIVA API data is also incomplete. Our Engineering is working on further recovery and the files will be delivered when data is complete and included in the reports.

Incomplete MQD/DIVA API and Bank Files Data

Oct <var data-var='date'>21</var>, <var data-var='time'>04:13</var> PDT Resolved - As of 12:30am PT, 7:30am GMT today, all historic webhooks have been cleared from the queues.Oct <var data-var='date'>20</var>, <var data-var='time'>18:22</var> PDT Update - At 15:01 PT, AWS confirmed that service restoration from their side after network connectivity issues they had across US-EAST-1 Regions was resolved. However, some AWS services are still degraded due to AWS's backlog of messages that will take a few hours to be processed, and an RCA will be shared from AWS side. All of Marqeta’s impacted services have been restored. For Webhooks service, processing was disrupted starting at approximately 12:00 AM PT. We have since restored service as of 12:30 PM PT, and we began flushing the queued webhooks starting at 6:15 PM PT, with processing expected to complete within 12 hours. We are actively monitoring systems closely to ensure continued stability.Oct <var data-var='date'>20</var>, <var data-var='time'>14:34</var> PDT Update - Numerous AWS services are degraded. This directly impacts several Marqeta services, including: Impacted services : Delay in all bank files and all DIVA MQD reports; Delay in recon and settlement reports; Delay in VAU and ABU reports; Delay in some direct deposits; Delay in fulfillment files; Partial outage to KYC Services; Degraded 3DS service; Degraded Sandbox service; Delay in webhooks; Recovered services : Network Updates service Tokenization services Core API response AWS reports they are continuing working on mitigating the issue and investigating the root cause for the network connectivity issues that are impacting AWS services. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has fully recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>13:35</var> PDT Update - Numerous AWS services are degraded. This directly impacts several Marqeta services, including: Impacted services : Delayed or unsent webhooks; Delay in all bank files and all DIVA MQD reports; Delay in recon and settlement reports; Delay in API responses; Delay in VAU and ABU reports; Delay in some direct deposits; Delay in fulfillment files; Partial outage to KYC Services; Degraded 3DS service; Degraded performance in tokenization functions. Degraded Sandbox service; Recovered /improved services : Network Updates service (Recovered). AWS reports they are continuing working on mitigating the issue and investigating the root cause for the network connectivity issues that are impacting AWS services. No ETA provided yet from AWS side for full service restoration. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has fully recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>12:34</var> PDT Update - Numerous AWS services are still degraded. This directly impacts several Marqeta services, including: Delayed or unsent webhooks; Delay in all bank files and all DIVA MQD reports; Delay in recon and settlement reports; Delay in API responses; Delay in VAU and ABU reports; Delay in some direct deposits; Delay in fulfillment files; Partial outage to KYC Services; Degraded Network Updates service; Degraded performance in tokenization functions. AWS reports they are continuing working on mitigating the issue and investigating the root cause for the network connectivity issues that are impacting AWS services. No ETA provided yet from AWS side for full service restoration. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has fully recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>11:41</var> PDT Update - We are continuing to monitor for any further issues.Oct <var data-var='date'>20</var>, <var data-var='time'>11:35</var> PDT Update - Numerous AWS services are still degraded. This directly impacts several Marqeta services, including: Delayed or unsent webhooks; Delay in all bank files and all DIVA MQD reports; Delay in recon and settlement reports; Delay in API responses; Delay in VAU and ABU reports; Delay in some direct deposits; Delay in fulfillment files; Partial outage to KYC Services; Degraded Network Updates service; Degraded performance in tokenization functions. AWS reports they are continuing working on mitigating the issue and investigating the root cause for the network connectivity issues that are impacting AWS services. No ETA provided yet from AWS side for full service restoration. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>10:35</var> PDT Update - Numerous AWS services are still degraded. This directly impacts several Marqeta services, including: Delayed or unsent webhooks; Delay in all bank files and all DIVA MQD reports; Delay in recon and settlement reports; Delay in API responses; Delay in VAU and ABU reports; Delay in some direct deposits; Delay in fulfillment files; Partial outage to KYC Services; Degraded Network Updates service; Degraded performance in tokenization functions. AWS reports they are continuing to investigate the root cause for the network connectivity issues that are impacting AWS services. They have not yet provided an ETA to partial or complete restoration, but are making periodic status updates. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>09:19</var> PDT Update - Numerous AWS services are still degraded. This directly impacts several Marqeta services, including: Delayed or unsent webhooks; Delay in all bank files and all DIVA MQD reports; Delay in recon and settlement reports; Delay in API responses; Delay in VAU and ABU reports; Delay in some direct deposits; Delay in fulfillment files; Partial outage to KYC Services; Degraded Network Updates service; Degraded performance in tokenization functions. AWS reports they are continuing to investigate the root cause for the network connectivity issues that are impacting AWS services. They have not yet provided an ETA to partial or complete restoration, but are making periodic status updates. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>08:22</var> PDT Update - Numerous AWS services are still degraded. This directly impacts several Marqeta services, including: Delayed or unsent webhooks; Delay in all bank files and all DIVA MQD reports; Delay in recon and settlement reports; Delay in API responses; Delay in VAU and ABU reports; Delay in some direct deposits; Delay in fulfillment files; Partial outage to KYC Services; Degraded Network Updates service; Degraded performance in tokenization functions. AWS reports they are continuing to investigate the root cause for the network connectivity issues that are impacting AWS services. They have not yet provided an ETA to partial or complete restoration, but are making periodic status updates. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>07:10</var> PDT Update - Numerous AWS services are still degraded. This directly impacts several Marqeta services, including: Delayed or unsent webhooks; Delay in all bank files and all DIVA MQD reports; Delay in recon and settlement reports; Delay in API responses; Delay in VAU and ABU reports; Delay in some direct deposits; Partial outage to KYC Services; Degraded performance in tokenization functions. AWS reports they are still experiencing elevated errors for new EC2 instance launches. They have not yet provided an ETA to partial or complete restoration, but are making periodic status updates. Engineers are continuing to monitor and will be able to resume affected services and reporting once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>06:51</var> PDT Update - Numerous AWS services are still affected, which directly impacts all webhooks. Marqeta engineers have also confirmed this will delay all bank files and all DIVA MQD reports. Additionally, recon and settlement reports are delayed. AWS reports they are still experiencing elevated errors for new EC2 instance launches. Engineers are continuing to monitor and will be able to slowly resume webhooks services and reporting once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>06:29</var> PDT Update - AWS Fargate is still affected, which directly impacts all webhooks. AWS reports they are continuing to work through processing the backlog of SQS messages in Lambda queues. Engineers are continuing to monitor and will slowly resume webhooks services once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>05:21</var> PDT Update - AWS Fargate is still affected, which directly impacts all webhooks. AWS reports working through processing the backlog of SQS messages in Lambda queues. Engineers are continuing to monitor and will slowly resume webhooks services once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>04:10</var> PDT Update - AWS reports they are seeing significant signs of recovery. AWS Fargate is still affected, which directly impacts all webhooks. Engineers are continuing to remediate and will slowly resume webhooks services once AWS has recovered.Oct <var data-var='date'>20</var>, <var data-var='time'>03:53</var> PDT Update - AWS reports they are seeing significant signs of recovery. We are monitoring this recovery and the recovery of Marqeta services.Oct <var data-var='date'>20</var>, <var data-var='time'>02:52</var> PDT Monitoring - AWS reports they are seeing significant signs of recovery. We are monitoring this recovery and the recovery of Marqeta services.Oct <var data-var='date'>20</var>, <var data-var='time'>02:06</var> PDT Identified - Marqeta engineers are tracking a confirmed AWS us-east-1 outage that is affecting multiple systems and other services. This can include SMS, Tokenization, KYC, APIs, Marqeta Dashboard, and more. Investigation is continuing to understand the scope of impacted systems. We are monitoring AWS communication channels for updates.Oct <var data-var='date'>20</var>, <var data-var='time'>01:12</var> PDT Update - Marqeta engineers are tracking a confirmed aws-us-east-1 outage that is affecting multiple systems and other services. This can include SMS, Tokenization, KYC, APIs, Marqeta Dashboard, and more. Investigation is underway and more information will be available soon.Oct <var data-var='date'>20</var>, <var data-var='time'>01:02</var> PDT Investigating - Marqeta engineers are tracking a possible aws-us-east-1 outage that is affecting multiple systems and other services. This can include SMS, tokenization, and more. Investigation is underway and more information will be available soon.

AWS us-east-1 outage

Oct <var data-var='date'>18</var>, <var data-var='time'>13:27</var> PDT Resolved - Visa has confirmed an event within their network which resulted in unexpected authorization failures. They have resolved the challenge and the Visa Network is now operating as expected. This incident is resolved.Oct <var data-var='date'>18</var>, <var data-var='time'>12:47</var> PDT Monitoring - Marqeta Engineers were alerted to an increase in STIPs impacting our customers on the Visa Network. Our monitors indicate that the STIPs have subsided. A response team is working with Visa to isolate the source of the challenge.

Is Marqeta down?

Marqeta status is UP

Lastest published events

Intermittent STIPs across multiple networks

Increase Rate Of Magstripe Declines

Delay in bank file delivery

Marqeta Services, Components and Regions

Recent Marqeta events

Intermittent STIPs across multiple networks

Increase Rate Of Magstripe Declines

Delay in bank file delivery

STIPS on Master Card network

FEDACH Processing Delay

Marqeta Dashboard Login Outage

Get your FREE account now,
5 monitors included!

Is Marqeta down?

Marqeta status is UP

Lastest published events

Intermittent STIPs across multiple networks

Increase Rate Of Magstripe Declines

Delay in bank file delivery

Marqeta Services, Components and Regions

Recent Marqeta events

Intermittent STIPs across multiple networks

Increase Rate Of Magstripe Declines

Delay in bank file delivery

STIPS on Master Card network

FEDACH Processing Delay

Marqeta Dashboard Login Outage

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!