May <var data-var='date'>21</var>, <var data-var='time'>07:32</var> PDT Resolved - Maintenance is complete. May <var data-var='date'>13</var>, <var data-var='time'>10:04</var> PDT Identified - We're making updates to how our private Sandbox environment handles and directs network traffic as part of ongoing infrastructure improvements. We expect this maintenance to be complete by May 22nd. Your users may lose the ability to interact with your Marqeta private sandbox. If so, please reach out to support@marqeta.com.

Private Sandbox Maintenance

May <var data-var='date'>20</var>, <var data-var='time'>01:00</var> PDT Resolved - Between 12:59 AM and 03:23 AM PDT, an external Mastercard incident caused intermittent errors for customers utilizing MDES token lifecycle actions. This resulted in temporary drops in token notifications and elevated 422 errors on the /v3/digitalwallettokentransitions endpoint due to upstream network issues. All services and webhook notifications have returned to baseline levels and are currently operating normally.

Mastercard MDES Lifecycle Service Degradation

May <var data-var='date'>15</var>, <var data-var='time'>20:46</var> PDT Resolved - We have successfully processed the final batch of clearing files. Following a thorough reconciliation against the Visa network summary, we can confirm that all expected clearing data is aligned and validated. May <var data-var='date'>15</var>, <var data-var='time'>16:58</var> PDT Update - Our teams have been actively monitoring file receipt and processing throughout the day, and we have confirmed that overall clearing volume today exceeds our previous day's totals, with most recent file processed at 14:35 PDT, We are also working to formally reconcile received data against the Visa network summary to provide full confirmation that all expected clearing activity has been recovered, and will provide a further update once reconciliation is complete. We sincerely apologize for any disruption this delay may have caused. May <var data-var='date'>15</var>, <var data-var='time'>10:16</var> PDT Identified - Our technical teams have confirmed after engaging Visa that we have processed from our side all files sent so far from Visa. Marqeta is currently addressing a potential delay in the delivery of clearing files from the Visa network., While these files are now being ingested and prioritized through our systems to restore full data visibility, customers may experience temporary discrepancies in data reporting or reconciliation against VSS reports for the impacted date. We are closely monitoring the processing of files sent from Visa side to ensure all records are accurately reflected, We will provide a further update as soon as the processing is complete and data integrity is fully verified. May <var data-var='date'>15</var>, <var data-var='time'>02:12</var> PDT Investigating - We receive 6 large files from Visa each day containing clearing for Marqeta customers. The last clearing file from the 14th of May has not yet been received from Visa. Impacted customers will have a lower number than normal of clearing webhooks and VSS reports particular to this file are delayed. We have reached out to Visa to understand what has happened, and to ensure the file is sent. Once the file has been received by Marqeta, it will be immediately processed and we will update this incident.

Visa clearing delayed for some customers.

May <var data-var='date'>12</var>, <var data-var='time'>20:10</var> PDT Resolved - Following a successful monitoring period, Marqeta systems have maintained stable processing since May 12, 08:40 AM PT with no further anomalies detected. We apologize for any impact this may have had on cardholders and appreciate your patience throughout the resolution process. May <var data-var='date'>12</var>, <var data-var='time'>09:23</var> PDT Monitoring - Following up on our previous updates, Pulse has informed us that the responsible acquirer has implemented a change to resolve the root cause, and Marqeta has cross-verified that (STIP) levels returned to normal parameters as of 08:40 AM PT. Our engineering teams will continue to closely observe the stability of the Pulse network to ensure no further disruptions occur for your cardholders, and we will provide a final resolution summary once the monitoring period is successfully complete. May <var data-var='date'>12</var>, <var data-var='time'>03:20</var> PDT Update - Our joint technical investigation with the Pulse team has successfully isolated the source of the anomalous data to a single acquiring partner. Pulse is actively working to engage the responsible acquirer to secure a path for remediation. Concurrently, Marqeta is escalating the issue with utmost priority. We will provide an update as soon as we have confirmation of the fix and an estimated time to resolution from Pulse. May <var data-var='date'>12</var>, <var data-var='time'>01:45</var> PDT Update - To keep our customers updated , we have initiated a joint T.S session with Pulse team, and we are working jointly to investigate the root cause and secure a path for remediation. We will provide an update as soon as we have a confirmed fix or an estimated time to resolution. May <var data-var='date'>12</var>, <var data-var='time'>00:48</var> PDT Identified - Marqeta Engineers have observed elevated STIPs on the Pulse network impacting some cardholders utilizing the Pulse network starting on 11th of May at 10:20 PM PDT. A dedicated response team has been assembled and has identified the technical issue as a processing failure within our systems when encountering a new data pattern originating from the Pulse network. To mitigate the impact and confirm the necessary fix, we have engaged the Pulse team for clarity on this new data behavior, and Pulse has raised an incident from their side to investigate. We will provide an update as soon as we have confirmation of the fix and an estimated time to resolution."

Elevated STIPs on the Pulse Network

May <var data-var='date'>11</var>, <var data-var='time'>15:57</var> PDT Resolved - All of Marqeta's services should now be fully operational. May <var data-var='date'>11</var>, <var data-var='time'>07:09</var> PDT Update - Nearly all Marqeta services are now fully operational. Engineers are continuing to mitigate some data discrepancies in certain services. We will provide our next update when the data has been fully remediated, or if we become aware of a substantial change in condition. May <var data-var='date'> 8</var>, <var data-var='time'>14:46</var> PDT Update - Nearly all Marqeta services are now fully operational. Engineers are continuing to mitigate some data discrepancies in certain services. We will continue monitoring the situation throughout the weekend. We will provide our next update Monday the 11th at 10:00 am EST, or if we become aware of a substantial change in condition. May <var data-var='date'> 8</var>, <var data-var='time'>07:00</var> PDT Update - We are continuing to observe improvements to Marqeta's services. Engineers are continuing to put mitigations in place, such as moving traffic away from the affected region and availability zone. We will continue monitoring the situation throughout the day for any changes in status. We will provide our next update when we receive additional information from AWS. May <var data-var='date'> 7</var>, <var data-var='time'>23:53</var> PDT Monitoring - We are continuing to observe improvements to Marqeta's services. Engineers are continuing to put mitigations in place, such as moving traffic away from the affected region and availability zone. We will also continue monitoring the situation overnight for any changes in status. Our next update will be at 10:00 EST. May <var data-var='date'> 7</var>, <var data-var='time'>22:29</var> PDT Update - AWS is observing some early signs of recovery. Some of Marqeta services remain impacted by the ongoing AWS us-east-1 outage. Our engineers are diligently monitoring the situation and implementing measures to reduce impact where possible. We will provide our next update when we receive additional information from AWS. May <var data-var='date'> 7</var>, <var data-var='time'>21:42</var> PDT Update - Marqeta services remain impacted by the ongoing AWS us-east-1 outage. Our engineers are diligently monitoring the situation and implementing measures to reduce impact where possible. We will provide our next update when we receive additional information from AWS. May <var data-var='date'> 7</var>, <var data-var='time'>21:14</var> PDT Update - Marqeta services remain impacted by the ongoing AWS us-east-1 outage. AWS is shifting traffic away from the impacted zone for additional services. Our teams continue monitoring the situation closely and will provide the next update in 30 minutes or when new information is available. May <var data-var='date'> 7</var>, <var data-var='time'>20:44</var> PDT Update - Marqeta services remain impacted by the ongoing AWS us-east-1 outage. AWS is shifting traffic away from the impacted zone for additional services. Our teams continue monitoring the situation closely and will provide the next update in 30 minutes or when new information is available. May <var data-var='date'> 7</var>, <var data-var='time'>20:14</var> PDT Update - Marqeta services remain impacted by the ongoing AWS us-east-1 outage. AWS is shifting traffic away from the impacted zone for additional services. Our teams continue monitoring the situation closely and will provide the next update in 30 minutes or when new information is available. May <var data-var='date'> 7</var>, <var data-var='time'>19:44</var> PDT Update - Marqeta services remain impacted by the ongoing AWS us-east-1 outage. AWS continues working to resolve the thermal issue in the affected availability zone. We are closely monitoring their progress and our service recovery. Next update in 30 minutes or sooner if conditions change. May <var data-var='date'> 7</var>, <var data-var='time'>19:14</var> PDT Update - Marqeta services remain impacted by the ongoing AWS us-east-1 outage. AWS continues working to resolve increased temperatures in the affected availability zone with no current ETA for full resolution. Some services are beginning to see improvement. Next update in 30 minutes or sooner if conditions change. May <var data-var='date'> 7</var>, <var data-var='time'>18:45</var> PDT Update - All Marqeta services remain impacted by the ongoing AWS us-east-1 outage with no ETA for resolution from AWS. The next update will be in 30 minutes or sooner if conditions change. May <var data-var='date'> 7</var>, <var data-var='time'>18:14</var> PDT Identified - We are continuing to monitor the AWS problem in us-east-1, and we are seeing impact across many Marqeta services due to this. We are actively removing traffic from this region to limit the impact to our customers. Next update will be in 30 minutes or when AWS provides any additional context. May <var data-var='date'> 7</var>, <var data-var='time'>17:46</var> PDT Investigating - Marqeta engineers are detecting an elevated number of STIPs starting from about 4:30PM PST. Teams are monitoring an AWS us-east-1 outage as the likely cause. We will provide updates every 30 minutes or upon discovery of new information.

AWS Outage Impacting Marqeta Services

May <var data-var='date'> 7</var>, <var data-var='time'>17:40</var> PDT Resolved - All replicated assets in the system should now be fully synchronized. May <var data-var='date'> 5</var>, <var data-var='time'>14:12</var> PDT Update - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is impacting data replication for transactional data in Diva reports and MQD endpoints. All replicated assets in the system are now fully synchronized with the exception of one cluster, which currently shows approximately 8 hours behind and is progressing toward full synchronization. Marqeta teams continue to monitor the recovery process and maintain engagement with the external vendor. More detailed information will be provided tomorrow morning, or if new information becomes available. May <var data-var='date'> 5</var>, <var data-var='time'>13:30</var> PDT Update - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams have observed assets continuing to improve. Some corrected reports will be available in the early morning of May 6th. We are continuing our engagement with the external vendor to resolve quicker and receive an ETA for full restoration. Our next update will be in 60 minutes. May <var data-var='date'> 5</var>, <var data-var='time'>12:30</var> PDT Update - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams have observed assets beginning to catch up. We are continuing our engagement with the external vendor to resolve quicker and receive an ETA for full restoration. Our next update will be in 60 minutes. May <var data-var='date'> 5</var>, <var data-var='time'>11:30</var> PDT Identified - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams have observed assets beginning to catch up. We are continuing our engagement with the external vendor to resolve quicker and receive an ETA for full restoration. Our next update will be in 60 minutes. May <var data-var='date'> 5</var>, <var data-var='time'>10:30</var> PDT Update - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams have escalated our engagement with the external vendor to resolve quicker. We will have more information about an ETA to restoration once the external vendor provides their own ETA. Our next update will be in 60 minutes. May <var data-var='date'> 5</var>, <var data-var='time'>09:32</var> PDT Update - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams have escalated our engagement with the external vendor to resolve quicker. We will have more information about an ETA to restoration once the external vendor provides their own ETA. Our next update will be in 60 minutes. May <var data-var='date'> 5</var>, <var data-var='time'>08:29</var> PDT Update - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams have escalated our engagement with the external vendor to resolve quicker. We will have more information about an ETA to restoration once the external vendor provides their own ETA. Our next update will be in 60 minutes. May <var data-var='date'> 5</var>, <var data-var='time'>07:29</var> PDT Update - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams have escalated our engagement with the external vendor to resolve quicker. We will have more information about an ETA to restoration once the external vendor provides their own ETA. Our next update will be in 60 minutes. May <var data-var='date'> 5</var>, <var data-var='time'>06:29</var> PDT Update - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams have escalated our engagement with the external vendor to resolve quicker. We will have more information about an ETA to restoration once the external vendor provides their own ETA. Our next update will be in 60 minutes. May <var data-var='date'> 5</var>, <var data-var='time'>05:29</var> PDT Investigating - An external vendor's unexpected cluster maintenance starting from about 4:00 PM PST on May 4th is causing a severe lag in data replication for transactional data in Diva reports and MQD endpoints. Marqeta teams are engaging with the external vendor to resolve the upstream cluster issue, running checks to quantify the data lag load times, and ensuring internal pipelines are ready to automatically process all delayed data once restored. We will have more information about an ETA to restoration once the external vendor provides their own ETA. Our next update will be in 60 minutes.

Delay in Diva reports and MQD endpoints

Apr <var data-var='date'>30</var>, <var data-var='time'>10:34</var> PDT Resolved - This incident has been resolved. Service has been restored and customers should no longer experience any impact. Apr <var data-var='date'>29</var>, <var data-var='time'>15:20</var> PDT Update - Engineers have confirmed that the incident is now stabilized. We will provide a final update once full resolution is confirmed Apr <var data-var='date'>29</var>, <var data-var='time'>14:15</var> PDT Update - Work continues toward a full resolution. We will provide the next update when new information becomes available. Apr <var data-var='date'>29</var>, <var data-var='time'>13:08</var> PDT Update - Our engineering teams remain focused on implementing the resolution. We will provide another update in approximately 60 minutes or when new information is provided. Apr <var data-var='date'>29</var>, <var data-var='time'>12:04</var> PDT Identified - Engineers have identified the root cause and are now working on a resolution. We will provide another update in approximately 60 minutes or when new information becomes available. Apr <var data-var='date'>29</var>, <var data-var='time'>11:35</var> PDT Update - Multiple teams are working together to investigate and resolve the 3DS performance degradation. Challenge timeouts and STIPs may continue to impact customers. We will provide another update in approximately 30 minutes. Apr <var data-var='date'>29</var>, <var data-var='time'>11:04</var> PDT Update - Our engineering team is actively continuing to investigate the root cause of the 3DS performance issue. Challenge timeouts and STIPs may still be impacting customers. We remain focused on resolution and will provide another update in approximately 30 minutes. Apr <var data-var='date'>29</var>, <var data-var='time'>10:32</var> PDT Update - 3DS is experiencing degraded performance that may cause challenge timeouts and STIPs. Our engineering team is investigating the root cause. Next update will be in approximately 30 minutes or when new information becomes available. Apr <var data-var='date'>29</var>, <var data-var='time'>10:04</var> PDT Investigating - Starting at 13:00 EST, 3DS is experiencing degraded performance that may cause challenge timeouts. Our engineering team is investigating the root cause. We will provide another update in approximately 30 minutes.

3DS Service Degradation

Apr <var data-var='date'> 6</var>, <var data-var='time'>11:32</var> PDT Resolved - Following an extended period of stability, we have verified that the tactical fix applied to our database environment has fully resolved the connectivity issues and all affected services are performing at baseline levels. We appreciate your patience throughout this event. Apr <var data-var='date'> 2</var>, <var data-var='time'>20:10</var> PDT Monitoring - Following our investigation with the database vendor, a tactical fix was successfully applied at 16:26 PST. Since the implementation of this fix, we have observed a significant improvement in database connection stability and a return to normal service levels. Our engineering team is continuously monitoring the environment to verify the long-term effectiveness of the resolution and ensure overall system stability. We will provide a final update once we have confirmed the issue is fully resolved and appreciate your continued patience. Apr <var data-var='date'> 2</var>, <var data-var='time'>13:31</var> PDT Identified - We have observed a database connectivity issue that intermittently impacted Marqeta services on April 1st between 16:04 and 17:56 PST. During this window, some customers may have experienced an elevated amount of STIPs as well as partial disruptions to tokenization and API traffic. Our engineering team has traced the instability to an intermittent database connection failure and has escalated the matter to our third-party vendor for support. We are currently engaged in an active troubleshooting with the vendor's support to isolate the root cause and prevent recurrence by applying a tactical solution. We apologize for any inconvenience this incident may caused, and will periodically provide updates once available.

Intermittent STIPs across multiple networks

Apr <var data-var='date'> 3</var>, <var data-var='time'>21:15</var> PDT Resolved - Engineers have resolved the last known instances of this issue. Apr <var data-var='date'> 3</var>, <var data-var='time'>19:55</var> PDT Update - Engineers are continuing to work to resolve the last few instances of this issue. We will update here once this work is complete. Apr <var data-var='date'> 3</var>, <var data-var='time'>12:44</var> PDT Identified - From March 24th, we are experiencing an increase in the rate of declines for card-present transactions using magnetic stripe swipe (magstripe). This may cause cardholder's physical cards to be declined even though the cards and accounts are in good standing. Engineers are working to resolve this issue today. We will update here once this work is complete.

Increase Rate Of Magstripe Declines

Mar <var data-var='date'>17</var>, <var data-var='time'>12:27</var> PDT Resolved - All bank files have been successfully delivered as of 12:17 PM PT. No further delays are expected. Mar <var data-var='date'>17</var>, <var data-var='time'>09:09</var> PDT Identified - We are currently experiencing delays in bank file delivery for some customers. Engineers have identified the issue and restarted the delivery process. All bank files are expected to be delivered within the next 3 hours.

Delay in bank file delivery

Mar <var data-var='date'>12</var>, <var data-var='time'>17:50</var> PDT Resolved - We are confirming that the Mastercard network disturbance, which caused two isolated periods of elevated (STIP) earlier today around 11:14 AM and 12:45 PM Pacific Time, is now fully resolved and the service is operating normally. Our engineering team monitored the service to ensure stability with no abnormality observed. Mar <var data-var='date'>12</var>, <var data-var='time'>13:29</var> PDT Monitoring - Marqeta Engineering team observed network disturbance resulting in STIPS on the Mastercard network, We observed two distinct periods of impact, starting around 11:14 AM and 12:45 PM Pacific Time , the status is currently stable and our team is actively engaged with MasterCard to understand the root cause. We are continuously monitoring the network to ensure stability.

STIPS on Master Card network

Mar <var data-var='date'> 4</var>, <var data-var='time'>09:05</var> PST Resolved - Processing has returned to its normal schedule. We are continuing to monitor the Federal Reserve Bank ACH Network for any reoccurrence and will provide updates if any new delays are identified. Mar <var data-var='date'> 3</var>, <var data-var='time'>18:04</var> PST Update - We have processed all ACH transactions received as of this time. However, according to the Federal Reserve Bank's Service Status, there may be continued delays affecting the next set of incoming transactions to the bank. We are continuing to monitor the ongoing issue at the Federal Reserve Bank ACH Network and will provide updates if any further delays are identified. Mar <var data-var='date'> 3</var>, <var data-var='time'>17:22</var> PST Update - We have processed all ACH transactions received as of this time. However, according to the Federal Reserve Bank's Service Status, there may be continued delays affecting the next set of incoming transactions to the bank. We are continuing to monitor the ongoing issue at the Federal Reserve Bank ACH Network and will provide updates if any further delays are identified. Mar <var data-var='date'> 3</var>, <var data-var='time'>13:32</var> PST Update - We are continuing to monitor the ongoing issue at Federal Reserve Bank ACH Network which may lead to a delay in the processing of ACH transactions. We have not yet been provided an ETA for resolution or any workarounds. We will update here as soon as they indicate that the issue has been resolved and is stable. Mar <var data-var='date'> 3</var>, <var data-var='time'>08:16</var> PST Investigating - Marqeta has been notified of an ongoing issue at Federal Reserve Bank ACH Network which may lead to a delay in the processing of ACH transactions. This can affect Direct Deposits, and ACH Origination. We have not yet been provided an ETA for resolution of their issue. We will update you as soon as they indicate that the issue has been resolved and is stable.

FEDACH Processing Delay

Feb <var data-var='date'>25</var>, <var data-var='time'>10:07</var> PST Resolved - At approximately 9:23 AM PT, Marqeta customers may have experienced issues logging into Marqeta Dashboard. Engineers quickly identified the root cause and implemented a fix. Customers can now access the Dashboard without issues.

Marqeta Dashboard Login Outage

Feb <var data-var='date'>13</var>, <var data-var='time'>14:25</var> PST Resolved - The full fix has now been successfully implemented across all Pulse routers as of February 13th approximately 2:15 PM PT. This follows the previously completed Visa rollback on February 10th, and the fix is now fully generalized across all networks. The implementation was executed in coordination with our network partners and completed as planned. We continue to closely monitor the environment and confirm that no duplicate transactions have been observed since Friday, February 6th at 6:00 PM PT / 2:00 AM UTC. The system remains stable, and no additional impact is anticipated to card networks. We appreciate your patience throughout this process. Feb <var data-var='date'>11</var>, <var data-var='time'>09:34</var> PST Update - The full rollback for Visa was successfully completed in close coordination with the Visa team on February 10th at 9:30 PM PT, The implementation was executed smoothly, and no abnormality was observed. For Pulse, we are actively coordinating next steps with their team. For an aligned maintenance window for fix implementation to ensure proper timing and minimize any potential downtime. We continue to closely monitor the environment and confirm that no duplicate transactions have been observed since Friday, February 6th at 6:00 PM PT / 2:00 AM UTC. At this time, we do not anticipate additional impact to either the Visa or Pulse card networks. We will provide further updates as additional information becomes available and appreciate your continued patience. Feb <var data-var='date'>10</var>, <var data-var='time'>12:32</var> PST Update - A non-impact partial fix for the issue was applied to Pulse on February 9th at 06:30 PM PT. Our engineering team is currently working with priority on finalizing the full-scale fix, and we will provide an update as soon as it is ready for deployment. We continue to confirm that no duplicate transactions have been observed since Friday, February 6th at 6:00 PM PT/2:00 AM UTC. We do not expect any further impact to either the Visa or Pulse card networks. We deeply appreciate your patience and remain fully committed to providing explicit communication throughout the resolution process. Feb <var data-var='date'> 9</var>, <var data-var='time'>13:41</var> PST Update - Engineers are scheduling the fix for Visa and Pulse. No duplicate transactions have been observed since Friday, February 6th at 6:00 PM PT/2:00 AM UTC. We do not expect any further impact to either card network. Feb <var data-var='date'> 6</var>, <var data-var='time'>18:05</var> PST Update - Our team has identified the cause of duplicate transaction postings as an issue introduced in a recent release. We've successfully completed a rollback for Mastercard. As a precautionary measure, we're scheduling the Visa and Pulse fixes for Monday, February 9th to avoid any potential impact during the weekend. All duplicate transactions will be automatically reversed within 48 hours, and no action is required from affected customers. We will provide our next update on Monday unless new information becomes available sooner. Feb <var data-var='date'> 6</var>, <var data-var='time'>16:00</var> PST Identified - We've identified the cause of the duplicate transaction issue and are implementing a fix. Feb <var data-var='date'> 6</var>, <var data-var='time'>15:16</var> PST Update - Our team is conducting a thorough investigation to determine the root cause while working on a resolution. We will provide updates as soon as new information becomes available. Feb <var data-var='date'> 6</var>, <var data-var='time'>12:45</var> PST Investigating - Starting January 22nd, some customers may notice duplicate transaction postings. Our engineering team is actively working to resolve this issue. Affected customers do not need to take any action.

Duplicate transaction postings

Feb <var data-var='date'>13</var>, <var data-var='time'>06:35</var> PST Resolved - The issue causing elevated API timeouts between 2:00 PM – 3:22 PM PT (22:00 – 23:22 UTC) on Thursday, February 12th has been fully resolved. The root cause was identified and mitigated, and our engineering team has continued to monitor the environment closely. No further abnormalities have been observed. We do not anticipate any additional impact at this time. Thank you for your patience. Feb <var data-var='date'>12</var>, <var data-var='time'>17:15</var> PST Monitoring - Some Marqeta customers may have experienced elevated API timeouts between 2:00 PM - 3:22 PM PT (22:00 - 23:22 UTC) on Thursday, February 12th. The root cause has been identified, and we do not anticipate any further impact at this time.

Elevated API Timeouts

Feb <var data-var='date'>11</var>, <var data-var='time'>06:21</var> PST Resolved - This has now been resolved. Feb <var data-var='date'>11</var>, <var data-var='time'>01:51</var> PST Investigating - Affected customers using the 2.0 simulation suite will be unable to use the simulations/cardtransactions/ and simulations/directdeposits/ apis, receiving a 503 response. We are escalating to the Product owners, we will update when they have identified the issue.

Sandbox simulations/cardtransactions/ endpoint is reporting 503

Feb <var data-var='date'> 2</var>, <var data-var='time'>11:30</var> PST Resolved - Engineers have confirmed that the increase in STIPs occurred between 10:29 AM PT - 10:35 AM PT due to a networking issue in one region. It is now resolved. Feb <var data-var='date'> 2</var>, <var data-var='time'>11:06</var> PST Monitoring - Engineers have diverted traffic from the impacted area. Customers should no longer be impacted. We will continue to monitor the situation and provide updates when new information becomes available. Feb <var data-var='date'> 2</var>, <var data-var='time'>11:01</var> PST Investigating - At approximately 10:29 AM PT Marqeta engineers observed an elevated amount of STIPs. We are currently investigating the issue. The next update will be at 11:30 AM PT.

STIPs across all networks

Jan <var data-var='date'>27</var>, <var data-var='time'>10:06</var> PST Resolved - Engineers have confirmed the reverted deployment has restored functionality. Teams will continue working to understand the root cause of the degraded functionality. Jan <var data-var='date'>27</var>, <var data-var='time'>09:51</var> PST Investigating - Marqeta engineers have been informed that Tokenmanager in the EU is experiencing an outage. Engineers are reverting a deployment that should restore full functionality shortly.

Tokenmanager EU is Down

Jan <var data-var='date'>24</var>, <var data-var='time'>07:20</var> PST Resolved - As of 7:15 AM Pacific time all Clearing Files have been processed and delivered. Jan <var data-var='date'>23</var>, <var data-var='time'>23:43</var> PST Monitoring - Clearing processing is operating as expected as of 11:40 PM Pacific time. The earlier errored condition has created a backlog which is being processed at this time. Clearing files will be delivered as they are created and we expect all files to be delivered by 8 AM Pacific time Saturday, January 24th. Jan <var data-var='date'>23</var>, <var data-var='time'>23:12</var> PST Identified - At 3 PM Pacific time on January 23rd the Marqeta clearing process experienced an errored condition. This error has been resolved and the clearing process has begun. Clearing files for our customers will be delayed up to 8 hours as a result of the error.

Clearing files are delayed

Jan <var data-var='date'>22</var>, <var data-var='time'>15:12</var> PST Resolved - Marqeta experienced an outage today affecting multiple services. Once the team isolated the root cause to a networking configuration issue, a fix was deployed and all impacted services were fully restored. The recovery is holding across all services. If you have any questions related to this matter, please reach out to support@marqeta.com. Jan <var data-var='date'>22</var>, <var data-var='time'>12:27</var> PST Monitoring - The team has isolated root cause to a networking configuration issue. A fix has been deployed and services have recovered. The teams are continuing to isolate and address any lingering effects of the outage. Further updates will be provided here as they are determined. Jan <var data-var='date'>22</var>, <var data-var='time'>12:15</var> PST Update - Marqeta is seeing multiple services impacted. This will include 500-level API errors, Tokenization Failures, Payment Processing Delays, Dashboard and Reporting Issues, KYC and Customer Onboarding, and Webhooks Functionality. The team has isolated root cause to a networking configuration issue. A fix has been deployed and most services have recovered. The teams are isolating and addressing any subsequent effects of the outage. We will be able to provide an ETA for full restoration once we are confident the recovery is holding across all services and all subsequent effects are known and addressed. We will update next as soon as restoration is confirmed. Jan <var data-var='date'>22</var>, <var data-var='time'>11:45</var> PST Update - Marqeta is seeing multiple services impacted. This will include 500-level API errors, Tokenization Failures, Payment Processing Delays, Dashboard and Reporting Issues, KYC and Customer Onboarding, and Webhooks Functionality. The team has isolated root cause to a networking configuration issue. A fix has been deployed and we are seeing services recovering. The teams are isolating and addressing any subsequent effects of the outage. We will be able to provide an ETA for full restoration once we are confident the recovery is holding across all services. We will update next at 12:15 PST or as soon as restoration is confirmed. Jan <var data-var='date'>22</var>, <var data-var='time'>11:30</var> PST Update - Marqeta is seeing multiple services impacted. This will include 500-level API errors, Tokenization Failures, Payment Processing Delays, Dashboard and Reporting Issues, KYC and Customer Onboarding, and Webhooks Functionality. The team has isolated root cause to a networking configuration issue. A fix has been deployed and we are seeing services recovering. We will be able to provide an ETA for full restoration once we are confident the recovery is holding across all services. We will update next at 11:45 PST or as soon as restoration is confirmed. Jan <var data-var='date'>22</var>, <var data-var='time'>11:15</var> PST Identified - Marqeta is seeing multiple services impacted. This will include 500-level API errors, Tokenization Failures, Payment Processing Delays, Dashboard and Reporting Issues, KYC and Customer Onboarding, and Webhooks Functionality. We have multiple teams engaged and triaging. The team has isolated root cause to a networking configuration issue. A fix has been deployed and we are seeing signs of recovery. We will be able to provide an ETA for full restoration once we are confident the recovery is holding. We will update next at 11:30 PST or as soon as restoration is confirmed. Jan <var data-var='date'>22</var>, <var data-var='time'>11:00</var> PST Update - Marqeta is seeing multiple services impacted. We have multiple teams engaged and triaging. The team has isolated root cause to a networking configuration issue. We are evaluating resolution paths presently, but do not yet have an ETA for restoration. We will update next at 11:15 PST. Jan <var data-var='date'>22</var>, <var data-var='time'>10:45</var> PST Update - Marqeta is seeing multiple services impacted. We have multiple teams engaged and triaging. The team has isolated root cause to a networking configuration issue. We are evaluating resolution paths presently, but do not yet have an ETA for restoration. We will update next at 11:00 PST. Jan <var data-var='date'>22</var>, <var data-var='time'>10:30</var> PST Update - Marqeta is seeing multiple services impacted. We have multiple teams engaged and triaging. The team has isolated root cause to a networking configuration issue. We will update next at 10:45 PST. Jan <var data-var='date'>22</var>, <var data-var='time'>10:14</var> PST Update - Marqeta is seeing multiple services impacted. We have multiple teams engaged and triaging. We do not yet have root cause. We will update next at 10:30 PST. Jan <var data-var='date'>22</var>, <var data-var='time'>09:56</var> PST Investigating - We have identified multiple outages across multiple services. Marqeta engineers are working hard to assess the scope of the outage, determine root cause, and provide timelines for restoration. Next update will be at 10:15 PST

Service Impact To Multiple Services

Jan <var data-var='date'>19</var>, <var data-var='time'>09:59</var> PST Resolved - We have implemented mitigation steps and see no further recurrence of this incident since those changes were implemented. Jan <var data-var='date'>19</var>, <var data-var='time'>05:19</var> PST Monitoring - The large spike in abnormal traffic has abated. We are still working to understand why this merchant sent so much traffic to us. Jan <var data-var='date'>19</var>, <var data-var='time'>04:50</var> PST Identified - Cardholders may have experienced elevated authentication declines, increased processing latency, and failed 3DS card authentications since 12:04 PM GMT/7:04 AM EST. The issue is caused by a large amount of abnormal traffic generated by a bad merchant. Our engineering teams identified and isolated the traffic source, and are working towards potential improvements.

3DS authentications impacted, 19th January

Jan <var data-var='date'> 8</var>, <var data-var='time'>12:22</var> PST Completed - The scheduled upgrade to our public sandbox databases has been successfully completed. Post-maintenance verifications have been performed, and system functionality has been confirmed. Thank you for your patience during this maintenance window. Jan <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'> 8</var>, <var data-var='time'>07:39</var> PST Update - We will be undergoing scheduled maintenance during this time. Jan <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> PST Update - We will be undergoing scheduled maintenance during this time. Jan <var data-var='date'> 8</var>, <var data-var='time'>11:00</var> PST Scheduled - Marqeta will be performing a scheduled upgrade to our public sandbox databases to enhance overall reliability and security. During the maintenance window, public sandbox environments may experience intermittent unavailability. The expected duration of this maintenance is approximately one hour. We appreciate your patience as we complete this work and will provide updates once available

Public Sandbox Database Upgrade

Jan <var data-var='date'> 7</var>, <var data-var='time'>02:01</var> PST Resolved - All clearings have been posted from the affected file. If you notice any discrepancies, please raised them with Marqeta support. Jan <var data-var='date'> 6</var>, <var data-var='time'>15:45</var> PST Update - The corrupted Visa file has been repaired and is now processing. Reporting should be completed shortly. We do not yet have an ETA for when transactions will be cleared, one will be shared here once it has been determined. Jan <var data-var='date'> 6</var>, <var data-var='time'>11:45</var> PST Update - Marqeta engineers have determined that Visa's corrupted file may not be possible to repair. We are continuing to wait for Visa's Clearing team to provide updates. We will update here once Visa sends a corrected file. We have no ETA as yet from Visa for this. Jan <var data-var='date'> 6</var>, <var data-var='time'>09:44</var> PST Update - We are continuing to wait for Visa's Clearing team to provide updates. In the meantime, Marqeta engineers are investigating the possibility of repairing Visa's corrupted file. We will update here once Visa sends a corrected file. We have no ETA as yet from Visa for this. We will also update here if the file repair approach is possible. Jan <var data-var='date'> 6</var>, <var data-var='time'>06:20</var> PST Update - Visa has escalated the issue to their Clearing team for investigation. Once their investigation in complete they will be able to provide us with an corrected file. Affected clearings will be processed once received. We will update here once Visa send a corrected file. We have no ETA as yet from Visa for this. Jan <var data-var='date'> 6</var>, <var data-var='time'>02:46</var> PST Identified - Some clearing transactions are delayed due to a corrupted file from Visa containing malformed characters. We've notified Visa and are waiting for a corrected file. Affected clearings will be processed once received. We will update here once Visa send a corrected file. We have no ETA as yet from Visa for this.

Delayed Visa Clearings January 6th

Dec <var data-var='date'>18</var>, <var data-var='time'>16:51</var> PST Resolved - As of 4:40 PM PT, engineers confirmed that all files have completed processing. Clearing webhooks and reporting should be functioning as expected. Dec <var data-var='date'>18</var>, <var data-var='time'>08:57</var> PST Update - For Mastercard, program level split files are continuing to progress with no ETA yet. Clearing webhooks have an ETA of one hour. Engineers are investigating reports of missing Mastercard T112 reports, and will look to resend them by end of day. While we don't have an exact ETA for complete resolution at this time, we will continue to provide additional details as they become available. Dec <var data-var='date'>17</var>, <var data-var='time'>17:36</var> PST Update - Visa files have finished processing. Engineers have completed the program level split files and they are now currently processing; we will continue to monitor their progress. While we don't have an exact ETA for complete resolution at this time, we will provide additional details as they become available. The next update will occur when all files have been fully processed and we can confirm operational status. Dec <var data-var='date'>17</var>, <var data-var='time'>08:20</var> PST Update - Engineers have clarified that Mastercard files were also impacted in the same timeframe. We are continuing to monitor the processing of Visa and Mastercard clearing files. Currently, the engineers have re-created the program level split files for clearing and reporting. As a part of this work there is a focus on preventing duplicate files from being sent. Once complete, we will send the split files to be cleared and/or delivered to customers. The next update will occur when we have confirmed that all files are caught up. Dec <var data-var='date'>16</var>, <var data-var='time'>03:17</var> PST Update - Engineers found the root cause, and have mitigated. The files are now being processed, Marqeta customers will start to see clearing webhooks now. We will monitor the procesing of clearing files, and update this incident when we are caught up, and will note that we are Operational then. Dec <var data-var='date'>16</var>, <var data-var='time'>02:51</var> PST Investigating - We are investigating issues processing Visa clearing and settlement files for today. We see no issues with clearings and settlement files for other networks. We see no impact to authorizations from any Network. Since the files have not been processed, impacted Marqeta customers on Visa will see an absence of clearing webhooks, and no Visa VSS report files until this incident is resolved. Prrefunded customers will see no clearings posted to cardholder balances until this incident is mitigated.

Clearings on Visa and Mastercard Delayed, 16 December

Nov <var data-var='date'>14</var>, <var data-var='time'>08:13</var> PST Resolved - Visa and Pulse reports should now be caught up for all customers. Visa clearing should now be caught up for all customers. Nov <var data-var='date'>13</var>, <var data-var='time'>10:50</var> PST Update - Pulse reports from today should be delivered for all customers. The Visa clearing is continuing to process. We now expect this to be caught up by tomorrow morning. Nov <var data-var='date'>13</var>, <var data-var='time'>09:49</var> PST Update - Visa reports are processed and delivery should be caught up for all customers. For Pulse the reports from before November 13th have been delivered. Pulse reports from today are still processing. These files will be processed and delivered today. For clearing on Visa, we are still working through the backlog of the files before November 13th. We expect this to be caught up today. Nov <var data-var='date'>13</var>, <var data-var='time'>08:42</var> PST Update - Engineers have resolved the root cause affecting the processing of Visa and Pulse clearing and reports. Visa reports are processed and delivery should be caught up for all customers. For Pulse the reports from before November 13th have been delivered. Pulse reports from today are still processing. These files will be processed and delivered today. For clearing on Visa, we are still working through the backlog of the files before November 13th. We expect this to be caught up today. Nov <var data-var='date'>13</var>, <var data-var='time'>07:22</var> PST Update - We are continuing to work on a fix for this issue. Full resolution is expected today. Nov <var data-var='date'>12</var>, <var data-var='time'>20:11</var> PST Update - We are continuing to work on a fix for this issue. Full resolution is expected tomorrow morning. Nov <var data-var='date'>12</var>, <var data-var='time'>17:12</var> PST Update - Engineering teams are actively working to restore file processing. We will continue to share updates and new information as it becomes available. Nov <var data-var='date'>12</var>, <var data-var='time'>14:39</var> PST Update - Engineering teams continue efforts to restore file processing capabilities. Additional updates and timelines to be communicated as they become available. Nov <var data-var='date'>12</var>, <var data-var='time'>12:36</var> PST Update - Engineers are continuing to work to restore file processing. They anticipate this will be resolved today. Additional updates and time frames will be provided here once they are known. Nov <var data-var='date'>12</var>, <var data-var='time'>11:32</var> PST Update - Engineers have confirmed that Pulse files are also affected. Engineers are continuing to actively investigate to mitigate the impact and resolve the issue. Nov <var data-var='date'>12</var>, <var data-var='time'>10:03</var> PST Identified - Issue was identified to be an internal service traffic issue that is impacting some of the file processing on Visa network, and Engineering team is currently working on developing a fix to address the reporting and clearing services issue. In the mean time, we are processing existing visa network files at reduced capacity. Nov <var data-var='date'>12</var>, <var data-var='time'>07:35</var> PST Investigating - On Nov 11, Marqeta observed a technical issue that is disturbing the communication of some of the clearing & settlement files on Visa Network. Our teams are actively investigating to mitigate the impact and resolve the issue.

Is Marqeta down?

Marqeta status is UP

Lastest published events

Private Sandbox Maintenance

Mastercard MDES Lifecycle Service Degradation

Visa clearing delayed for some customers.

Marqeta Services, Components and Regions

Recent Marqeta events

Private Sandbox Maintenance

Mastercard MDES Lifecycle Service Degradation

Visa clearing delayed for some customers.

Elevated STIPs on the Pulse Network

AWS Outage Impacting Marqeta Services

Delay in Diva reports and MQD endpoints

Get your FREE account now,
5 monitors included!

Is Marqeta down?

Marqeta status is UP

Lastest published events

Private Sandbox Maintenance

Mastercard MDES Lifecycle Service Degradation

Visa clearing delayed for some customers.

Marqeta Services, Components and Regions

Recent Marqeta events

Private Sandbox Maintenance

Mastercard MDES Lifecycle Service Degradation

Visa clearing delayed for some customers.

Elevated STIPs on the Pulse Network

AWS Outage Impacting Marqeta Services

Delay in Diva reports and MQD endpoints

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!