Apr <var data-var='date'> 2</var>, <var data-var='time'>17:18</var> UTC Resolved - This incident has been resolved. If you encounter any issues related to Google Ads, GA analytics, or Google Sheets/Drive, please follow the steps suggested in this incident.Apr <var data-var='date'> 2</var>, <var data-var='time'>13:53</var> UTC Monitoring - We've updated the credentials for our Google integration as part of a security rotation. If you have a Google Ads, Google Analytics, Google Sheets or Google Drive connection, you may need to reconnect it: - Go to your Connections page - Open the affected connection - Click Reconnect and follow the Google authorization prompt If you have any trouble, our support team is here to help.

Job Failures Related to Google-Related Services (Ads, Analytics, Sheets, Drive)

Apr <var data-var='date'> 2</var>, <var data-var='time'>13:45</var> UTC Resolved - The incident has been resolved.Apr <var data-var='date'> 2</var>, <var data-var='time'>12:29</var> UTC Identified - The issue has been identified and a fix has been implemented. Please rerun the jobs on a newly created cluster.Apr <var data-var='date'> 2</var>, <var data-var='time'>11:50</var> UTC Investigating - We are currently investigating this issue

Job Failures Related to Snowflake Connections

Apr <var data-var='date'> 2</var>, <var data-var='time'>11:50</var> UTC Resolved - This incident has been resolved.Feb <var data-var='date'>17</var>, <var data-var='time'>03:48</var> UTC Monitoring - A fix has been implemented, and we are actively monitoring the results to ensure stability. Please proceed with running the jobs on a newly created cluster. This issue impacts packages that include the ExecuteSQL component and BigQuery V1. We apologize for the inconvenience and appreciate your patience.Feb <var data-var='date'> 9</var>, <var data-var='time'>16:18</var> UTC Identified - We have identified an issue causing job failures with the BigQuery V1 connector. This was introduced by a recent library upgrade that resulted in a regression affecting the connector. We are actively working on a fix. In the meantime, customers may migrate to the BigQuery V2 connector by following our documentation or by reaching out to our support team for assistance. We apologize for the inconvenience and appreciate your patience.

BigQuery V1 Job Failures

Feb <var data-var='date'>17</var>, <var data-var='time'>03:39</var> UTC Resolved - The incident has been resolved.Feb <var data-var='date'>10</var>, <var data-var='time'>19:08</var> UTC Monitoring - A fix has been implemented by our upstream provider and services are starting to recover. We are monitoring the results.Feb <var data-var='date'>10</var>, <var data-var='time'>17:24</var> UTC Identified - The issue has been identified with our upstream provider and a fix is being implemented.

Intermittent DNS Failures

Feb <var data-var='date'> 2</var>, <var data-var='time'>12:30</var> UTC Completed - The scheduled maintenance has been completed.Feb <var data-var='date'> 2</var>, <var data-var='time'>12:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Feb <var data-var='date'> 1</var>, <var data-var='time'>14:15</var> UTC Scheduled - On Feb 2, from 12:00 to 12:30 UTC, we will be performing required database maintenance. During this time, the platform will enter maintenance mode and the creation of new clusters and execution of jobs will be temporarily unavailable.

Required Database Maintenance

Jan <var data-var='date'>19</var>, <var data-var='time'>16:12</var> UTC Resolved - The incident has been resolved. We will be writing a post-mortem report soon.Jan <var data-var='date'>19</var>, <var data-var='time'>10:22</var> UTC Update - We are continuing to recover the processing engine workloads and jobs should now be working moving forward.Jan <var data-var='date'>19</var>, <var data-var='time'>10:03</var> UTC Identified - We’re seeing an improvement in our applications and currently working on restoring processing engine workloads.Jan <var data-var='date'>19</var>, <var data-var='time'>04:05</var> UTC Update - The issue has been identified with the upstream provided and we are continuing to work with them to fix.Jan <var data-var='date'>19</var>, <var data-var='time'>00:45</var> UTC Investigating - We are currently investigating the issue.

Some Clusters Stuck on Creating and Dashboard Inaccessible

Dec <var data-var='date'> 3</var>, <var data-var='time'>05:58</var> UTC Resolved - The incident has been resolved. The root cause was an issue with our upstream provider, which prevented clusters and jobs from triggering as expected. We are taking steps to improve our monitoring systems to ensure earlier detection in the future. We sincerely apologize for the inconvenience.Dec <var data-var='date'> 3</var>, <var data-var='time'>05:57</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.Dec <var data-var='date'> 3</var>, <var data-var='time'>04:18</var> UTC Investigating - We are currently investigating the issue.

Clusters Stuck on Creating, Jobs Not Running

Nov <var data-var='date'>21</var>, <var data-var='time'>07:00</var> UTC Resolved - Jobs are partially failed to be executed on the clusters due to an issue from our upstream providers. The issue was resolved at Nov 21, 07:30 UTC.

Partial Job Executions Failure

Oct <var data-var='date'>22</var>, <var data-var='time'>03:00</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>20</var>, <var data-var='time'>11:47</var> UTC Monitoring - Due to AWS's incident the following components are still impacted, SFTP To Go's audit logs and Cron To Go's processing engine For more details please see below AWS status- https://health.aws.amazon.com/health/status

Due to AWS's incident some components are still impacted

Sep <var data-var='date'>27</var>, <var data-var='time'>02:30</var> UTC Resolved - The incident has been resolved. The root cause was an issue with our upstream provider, which prevented clusters and jobs from triggering as expected. We are taking steps to improve our monitoring systems to ensure earlier detection in the future. We sincerely apologize for the inconvenience.Sep <var data-var='date'>27</var>, <var data-var='time'>01:27</var> UTC Investigating - We are currently investigating the issue.

Aug <var data-var='date'>14</var>, <var data-var='time'>09:52</var> UTC Resolved - The incident has been resolved.Aug <var data-var='date'>14</var>, <var data-var='time'>09:23</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Job workloads aren't affected and should still run as normal.Aug <var data-var='date'>14</var>, <var data-var='time'>08:12</var> UTC Investigating - We are currently investigating this issue.

Clusters Taking Longer Time to Provision and Terminate

Aug <var data-var='date'> 8</var>, <var data-var='time'>10:21</var> UTC Resolved - The incident has been resolved. The issue was caused by an issue on our upstream provider which caused clusters and jobs to not trigger. We will be working on improving our monitoring systems so we can detect the issue early. We apologize for the inconvenience caused.Aug <var data-var='date'> 8</var>, <var data-var='time'>08:24</var> UTC Monitoring - The issue has been identified and a fix has been implemented. We are currently monitoring.Aug <var data-var='date'> 8</var>, <var data-var='time'>08:13</var> UTC Investigating - We are currently investigating the issue.

Jul <var data-var='date'> 5</var>, <var data-var='time'>02:31</var> UTC Resolved - The incident has been resolved.Jul <var data-var='date'> 4</var>, <var data-var='time'>22:40</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. We apologize for the inconvenience caused.Jul <var data-var='date'> 4</var>, <var data-var='time'>22:32</var> UTC Investigating - We are currently investigating the issue.

Clusters Stuck on Creating

Jun <var data-var='date'>22</var>, <var data-var='time'>08:00</var> UTC Completed - The scheduled maintenance has been completed.Jun <var data-var='date'>22</var>, <var data-var='time'>07:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Jun <var data-var='date'>17</var>, <var data-var='time'>05:08</var> UTC Scheduled - During this maintenance, the platform will enter maintenance mode-- new clusters creation and jobs execution will be unavailable.

Database Maintenance

Jun <var data-var='date'>19</var>, <var data-var='time'>22:30</var> UTC Resolved - RCA: Initial findings: There seems to have been an issue with our upstream provider, which we're working with them to identify why. The above solution resolves the issue.Jun <var data-var='date'>19</var>, <var data-var='time'>21:47</var> UTC Monitoring - RCA: It looks like an issue one of our upstream providers. Follow the steps shared to resolve the issue. Detailed RCA will be shared soon.Jun <var data-var='date'>19</var>, <var data-var='time'>20:51</var> UTC Identified - The issue has been identified, and please find the fix below to resolve the connectivity issue. Please delete/refresh the entry on your known_hosts file. You can do either of the following: Delete the specific line on ~/.ssh/known_hosts file which contains virginia-tunnel.xplenty.com. Make sure to save. Execute ssh-keygen -R "[virginia-tunnel.xplenty.com]:50683" on the tunnel instanceJun <var data-var='date'>19</var>, <var data-var='time'>20:41</var> UTC Investigating - Hi All, We are experiencing an issue with our Reverse SSH connections, and the jobs are failing. Action: Investigating the issue

Reverse SSH connectivity issue

May <var data-var='date'>28</var>, <var data-var='time'>19:50</var> UTC Resolved - The incident has been resolved. We apologize for the inconvenience caused.May <var data-var='date'>28</var>, <var data-var='time'>19:26</var> UTC Monitoring - A fix has been implemented and we are monitoring the result. We apologize for the inconvenience caused here.May <var data-var='date'>28</var>, <var data-var='time'>18:03</var> UTC Investigating - We are currently investigating the issue which is causing the dashboard to return intermittent errors.

Intermittent Dashboard Errors

May <var data-var='date'> 5</var>, <var data-var='time'>01:34</var> UTC Resolved - The incident has been resolved.May <var data-var='date'> 5</var>, <var data-var='time'>01:31</var> UTC Monitoring - A fix has been implemented and we are monitoring the results.May <var data-var='date'> 5</var>, <var data-var='time'>01:26</var> UTC Investigating - We are currently investigating an issue which caused jobs to be stuck on pending.

Jobs Stuck on Pending

Apr <var data-var='date'>26</var>, <var data-var='time'>11:06</var> UTC Resolved - The incident has been resolved.Apr <var data-var='date'>25</var>, <var data-var='time'>16:58</var> UTC Monitoring - A fix has been implemented. Please reconnect your Bing Ads connection and rerun the jobs on a newly provisioned cluster.Apr <var data-var='date'>22</var>, <var data-var='time'>12:41</var> UTC Investigating - Customers may experience issues with jobs that read data from Bing Ads. Our engineers are investigating the issues.

Issues retrieving data from Bing Ads

Apr <var data-var='date'>11</var>, <var data-var='time'>15:30</var> UTC Resolved - Jobs are partially failed to be executed on the clusters due to an issue from our upstream providers. The issue was resolved at April 13, 12:30 UTC.

Mar <var data-var='date'>18</var>, <var data-var='time'>07:26</var> UTC Resolved - The incident has been resolved.Mar <var data-var='date'>18</var>, <var data-var='time'>06:21</var> UTC Monitoring - Clusters are now being created. We are continuing to monitor the issue.Mar <var data-var='date'>18</var>, <var data-var='time'>03:46</var> UTC Identified - We are seeing some clusters stuck on Creating on the Virginia region and have identified that it's due to an issue with our upstream provider. Will provide updates as soon as we have it.

Some Clusters Stuck on Creating

Feb <var data-var='date'>24</var>, <var data-var='time'>20:13</var> UTC Resolved - From Feb 24th, 14:30 UTC to 19:41 UTC, an issue with our job notification service caused jobs to skip the Running state and display a blank runtime. While Dataflow packages may have run normally (pending further confirmation), Workflows were affected, leading to job failures. The issue has now been resolved. If needed, please rerun the jobs or use "Run One-Off" on the schedule page. A post-mortem will be released soon. We sincerely apologize for the inconvenience.

Job Notification Issue Causing Job Failures

Jan <var data-var='date'>15</var>, <var data-var='time'>13:07</var> UTC Resolved - From 10:25 UTC to 13:00 UTC, job failures occurred in packages using the file storage destination component due to a faulty deployment. The issue has been resolved. We apologize for any inconvenience this may have caused.

Job Failures on File Storage Destination

Dec <var data-var='date'>21</var>, <var data-var='time'>00:45</var> UTC Resolved - This incident has been resolved.Dec <var data-var='date'>20</var>, <var data-var='time'>19:40</var> UTC Investigating - We are aware of an intermittent issue affecting the dashboard. If you encounter this problem, please try refreshing the page. Our development team is actively investigating the root cause and will implement a fix as soon as possible. We apologize for the inconvenience and appreciate your patience.

Dashboard Loading Issue

Dec <var data-var='date'>20</var>, <var data-var='time'>09:32</var> UTC Resolved - This incident has been resolved.Dec <var data-var='date'>20</var>, <var data-var='time'>09:20</var> UTC Investigating - Our API is currently down due to an emergency maintenance of our upstream provider. We will update shortly

API down due to an emergency maintenance of our upstream provider

Dec <var data-var='date'>12</var>, <var data-var='time'>21:25</var> UTC Resolved - There was an issue with our SSH server in the Oregon region, and the dev team applied the fix as of 21:40 UTC

Is Xplenty down?

Xplenty status is UP

Lastest published events

Job Failures Related to Google-Related Services (Ads, Analytics, Sheets, Drive)

Job Failures Related to Snowflake Connections

BigQuery V1 Job Failures

Xplenty Services, Components and Regions

Recent Xplenty events

Job Failures Related to Google-Related Services (Ads, Analytics, Sheets, Drive)

Job Failures Related to Snowflake Connections

BigQuery V1 Job Failures

Intermittent DNS Failures

Required Database Maintenance

Some Clusters Stuck on Creating and Dashboard Inaccessible

Get your FREE account now,
5 monitors included!

Is Xplenty down?

Xplenty status is UP

Lastest published events

Job Failures Related to Google-Related Services (Ads, Analytics, Sheets, Drive)

Job Failures Related to Snowflake Connections

BigQuery V1 Job Failures

Xplenty Services, Components and Regions

Recent Xplenty events

Job Failures Related to Google-Related Services (Ads, Analytics, Sheets, Drive)

Job Failures Related to Snowflake Connections

BigQuery V1 Job Failures

Intermittent DNS Failures

Required Database Maintenance

Some Clusters Stuck on Creating and Dashboard Inaccessible

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!