May <var data-var='date'>16</var>, <var data-var='time'>12:50</var> EDT Resolved - This incident has been resolved and we are continuing to monitor. We apologize for the inconvenience. May <var data-var='date'>16</var>, <var data-var='time'>12:40</var> EDT Identified - Our US environments are experiencing instability stemming from a maintenance task by an upstream provider. We are working to mitigate and will update shortly.

API incident due to cloud provider maintenance

May <var data-var='date'>16</var>, <var data-var='time'>12:04</var> EDT Resolved - This issue has been resolved and we are continuing to monitor. Apologies for the inconvenience. May <var data-var='date'>16</var>, <var data-var='time'>11:55</var> EDT Update - A fix has been implemented and we are monitoring. May <var data-var='date'>16</var>, <var data-var='time'>11:30</var> EDT Identified - The issue has been identified and we are working on a fix. May <var data-var='date'>16</var>, <var data-var='time'>11:28</var> EDT Investigating - We are currently investigating an issue and will update when we know more.

API incident in EU region

Jan <var data-var='date'>22</var>, <var data-var='time'>16:17</var> EST Resolved - This incident has been resolved. Jan <var data-var='date'>22</var>, <var data-var='time'>13:53</var> EST Monitoring - A fix has been implemented and we are monitoring to assure peak efficiency. Jan <var data-var='date'>22</var>, <var data-var='time'>12:47</var> EST Update - We are continuing to monitor recovery and will update as soon as we have further details. Jan <var data-var='date'>22</var>, <var data-var='time'>11:13</var> EST Update - We are continuing to monitor the fix as we recover. Jan <var data-var='date'>22</var>, <var data-var='time'>10:35</var> EST Update - We are currently implementing a fix for the issue. Jan <var data-var='date'>22</var>, <var data-var='time'>09:59</var> EST Identified - An issue has been identified and we are working on a fix. Jan <var data-var='date'>22</var>, <var data-var='time'>09:51</var> EST Investigating - We are currently investigating an incident affecting our EU environments. Logging into the Integration Hub and runs are affected. We are actively investigating and will update when we know more.

EU environments experiencing instability

Jun <var data-var='date'>12</var>, <var data-var='time'>17:21</var> EDT Resolved - This issue has been resolved and all services are caught up and fully operational. We will continue to monitor to ensure stability. Jun <var data-var='date'>12</var>, <var data-var='time'>16:50</var> EDT Monitoring - All systems are back online and we are monitoring the recovery of the backlog of jobs. Jun <var data-var='date'>12</var>, <var data-var='time'>15:43</var> EDT Update - EU environments have recovered and we are continuing to monitor. US environments remain affected by the outage. Jun <var data-var='date'>12</var>, <var data-var='time'>15:20</var> EDT Update - Runs are firing with delays in our EU environment. We are monitoring. Jun <var data-var='date'>12</var>, <var data-var='time'>14:37</var> EDT Update - The issue with our cloud hosting provider is ongoing. We are continuing to monitor. Jun <var data-var='date'>12</var>, <var data-var='time'>14:13</var> EDT Identified - All services are inaccessible or responding slowly due to an issue with our cloud hosting provider. We will update as soon as we know more.

Services inaccessable

Mar <var data-var='date'>21</var>, <var data-var='time'>10:24</var> EDT Resolved - This issue has been resolved and runs are processing normally. Mar <var data-var='date'>21</var>, <var data-var='time'>10:00</var> EDT Identified - We have identified the issue and deployed a fix. Mar <var data-var='date'>21</var>, <var data-var='time'>09:36</var> EDT Investigating - Some runs in the US region are firing with delays. We are investigating.

Some runs firing with delays

Nov <var data-var='date'> 1</var>, <var data-var='time'>19:13</var> EDT Resolved - This issue has been resolved. Nov <var data-var='date'> 1</var>, <var data-var='time'>19:09</var> EDT Monitoring - A fix has been implemented and we are monitoring recovery Nov <var data-var='date'> 1</var>, <var data-var='time'>19:03</var> EDT Identified - The issue has been identified and a fix is being implemented. Nov <var data-var='date'> 1</var>, <var data-var='time'>19:02</var> EDT Update - The issue with runs has been identified and we are implementing a fix. No other services are affected. Nov <var data-var='date'> 1</var>, <var data-var='time'>18:49</var> EDT Investigating - Runs for some US-Based accounts are processing with delays. We are investigating.

Runs processing with delays

Oct <var data-var='date'> 1</var>, <var data-var='time'>15:30</var> EDT Resolved - This incident has been resolved. Oct <var data-var='date'> 1</var>, <var data-var='time'>15:25</var> EDT Investigating - We are currently investigating an issue in which some runs are firing with delays.

Runs slowed

Sep <var data-var='date'>27</var>, <var data-var='time'>10:47</var> EDT Resolved - This incident has been resolved and we are continuing to monitor Sep <var data-var='date'>27</var>, <var data-var='time'>09:35</var> EDT Identified - The issue has been identified and we are working on a fix Sep <var data-var='date'>27</var>, <var data-var='time'>09:15</var> EDT Investigating - We are investigating.

Some runs processing with delays

Sep <var data-var='date'>26</var>, <var data-var='time'>15:52</var> EDT Resolved - This incident has been resolved. Sep <var data-var='date'>26</var>, <var data-var='time'>15:12</var> EDT Monitoring - A fix has been implemented and we are monitoring. Sep <var data-var='date'>26</var>, <var data-var='time'>13:35</var> EDT Identified - Runs are starting with delays as we investigate the underlying cause.

Sep <var data-var='date'>21</var>, <var data-var='time'>21:35</var> EDT Resolved - The underlying service issue causing instability has been resolved and runs are firing. We are continuing to monitor for the next few hours and will open a new incident if necessary. Sep <var data-var='date'>21</var>, <var data-var='time'>21:09</var> EDT Monitoring - A fix has been implemented and runs are recovering. We are closely monitoring and will update when fully resolved. Sep <var data-var='date'>21</var>, <var data-var='time'>19:35</var> EDT Update - The Pandium Integration Hub is fully operational however jobs are still running with delays. We will provide an update as soon as possible. Sep <var data-var='date'>21</var>, <var data-var='time'>16:03</var> EDT Update - We are continuing to work on a fix for this issue. Sep <var data-var='date'>21</var>, <var data-var='time'>14:36</var> EDT Identified - We have identified that there is an issue with our underlying cloud hosting provider and we are working with them to implement a fix. They have escalated this issue and we will provide an update as soon as possible. Sep <var data-var='date'>21</var>, <var data-var='time'>13:06</var> EDT Monitoring - A fix has been implemented and we are monitoring recovery. Sep <var data-var='date'>21</var>, <var data-var='time'>11:45</var> EDT Investigating - After recovery, we are experiencing a different issue with our underlying platform. We are investigating Sep <var data-var='date'>21</var>, <var data-var='time'>10:37</var> EDT Update - We are continuing to monitor for any further issues. Sep <var data-var='date'>21</var>, <var data-var='time'>10:09</var> EDT Monitoring - The released fix was effective and we are monitoring platform recovery. Sep <var data-var='date'>21</var>, <var data-var='time'>10:05</var> EDT Identified - The issue has been identified and a fix has been released Sep <var data-var='date'>21</var>, <var data-var='time'>09:35</var> EDT Update - We have identified the issue and are investigating a fix. Sep <var data-var='date'>21</var>, <var data-var='time'>09:02</var> EDT Update - We are continuing to investigate this issue. Sep <var data-var='date'>21</var>, <var data-var='time'>09:02</var> EDT Investigating - Some runs are firing slowly due to an intermittent issue with token management. We are investigating.

Runs degraded

Jul <var data-var='date'>25</var>, <var data-var='time'>02:39</var> EDT Resolved - Will Continuing to monitoring. All systems green. Jul <var data-var='date'>25</var>, <var data-var='time'>02:27</var> EDT Monitoring - We have increased replicas and are seeing a reduction in error counts. Monitoring. Jul <var data-var='date'>25</var>, <var data-var='time'>02:12</var> EDT Investigating - API responses exceeding error limits. We are investigating.

API responses exceeding error limits.

Jul <var data-var='date'>14</var>, <var data-var='time'>17:42</var> EDT Resolved - Scheduled run runs have caught up, and we have not experienced any more issues with control plane. Jul <var data-var='date'>14</var>, <var data-var='time'>13:37</var> EDT Monitoring - A fix has been implemented. We are now monitoring recovery. Jul <var data-var='date'>14</var>, <var data-var='time'>12:35</var> EDT Update - The underlying issue has been identified. Ectd cluster of the control plane is having a fit. Working on mitigation. Jul <var data-var='date'>14</var>, <var data-var='time'>11:53</var> EDT Identified - Issue is with the control plane of underlying hosting provider. We are working with them on mitigation. Jul <var data-var='date'>14</var>, <var data-var='time'>03:06</var> EDT Investigating - Underlying Control Plane under high load. Investigating. Run and API are running slow.

Degraded Performance

Jan <var data-var='date'> 6</var>, <var data-var='time'>08:30</var> EST Resolved - Runs were failing across the platform due to a service that was stuck in a terminating state.

Runs were failing due to inability to access secrets

Nov <var data-var='date'>15</var>, <var data-var='time'>11:00</var> EST Resolved - This incident has been resolved and we're monitoring to ensure performance. Nov <var data-var='date'>15</var>, <var data-var='time'>10:59</var> EST Update - Activity reports are slow to load via the UI. There is no issue with runs or integrations. Nov <var data-var='date'>15</var>, <var data-var='time'>10:45</var> EST Identified - Runs are currently processing slower than normal. We are currently addressing the issue and will update shortly.

Runs Degradation

Aug <var data-var='date'>12</var>, <var data-var='time'>10:31</var> EDT Resolved - The underlying issue that caused webhook runs to be queued is resolved. New manual, scheduled and webhook runs are all functioning normally while queued webhook payloads are being processed. Aug <var data-var='date'>12</var>, <var data-var='time'>09:46</var> EDT Monitoring - While a core system restarted, runs were saved and queued, including webhook payloads. They are currently processing and we expect this issue to resolve shortly. We are continuing to monitor.

Runs degraded after core system restart

Jun <var data-var='date'> 2</var>, <var data-var='time'>11:42</var> EDT Resolved - This incident has been resolved. Jun <var data-var='date'> 2</var>, <var data-var='time'>11:38</var> EDT Monitoring - We are monitoring a fix. Jun <var data-var='date'> 2</var>, <var data-var='time'>11:31</var> EDT Identified - We have identified an issue and are implementing a fix. Jun <var data-var='date'> 2</var>, <var data-var='time'>11:27</var> EDT Update - Some front-ends are unavailable and we are investigating. Runs are not affected. Jun <var data-var='date'> 2</var>, <var data-var='time'>11:25</var> EDT Investigating - We are investigating this issue.

Some APIs degraded

May <var data-var='date'>24</var>, <var data-var='time'>13:32</var> EDT Resolved - This issue has been resolved. May <var data-var='date'>24</var>, <var data-var='time'>13:26</var> EDT Monitoring - A fix has been implemented and we are monitoring. May <var data-var='date'>24</var>, <var data-var='time'>13:24</var> EDT Identified - We have identified an issue we believe is the cause for the API instability. May <var data-var='date'>24</var>, <var data-var='time'>13:22</var> EDT Update - Update: All jobs are continuing to run. May <var data-var='date'>24</var>, <var data-var='time'>13:19</var> EDT Investigating - We are experiencing intermittent issues with our APIs. We are investigating the cause.

Degraded APIs

May <var data-var='date'>24</var>, <var data-var='time'>09:29</var> EDT Resolved - This issue is resolved. May <var data-var='date'>24</var>, <var data-var='time'>09:21</var> EDT Monitoring - A fix has been implemented and we are monitoring May <var data-var='date'>24</var>, <var data-var='time'>09:20</var> EDT Investigating - We are currently investigating this issue. Runs are not affected.

Some marketplaces degraded

Apr <var data-var='date'>24</var>, <var data-var='time'>15:32</var> EDT Resolved - This incident has been resolved. Apr <var data-var='date'>24</var>, <var data-var='time'>15:17</var> EDT Monitoring - The API is recovering and we will continue to monitor. Apr <var data-var='date'>24</var>, <var data-var='time'>15:16</var> EDT Identified - Some pages may be slow to load. We are working to recover.

Apr <var data-var='date'>14</var>, <var data-var='time'>13:54</var> EDT Resolved - This issue has been resolved. Apr <var data-var='date'>14</var>, <var data-var='time'>13:53</var> EDT Monitoring - We identified an issue where some sites were inaccessible. We implemented a fix and are monitoring.

Some sites inaccessible

Feb <var data-var='date'>17</var>, <var data-var='time'>12:09</var> EST Resolved - This issue has been resolved. Feb <var data-var='date'>17</var>, <var data-var='time'>12:08</var> EST Update - We are continuing to investigate this issue. Feb <var data-var='date'>17</var>, <var data-var='time'>12:07</var> EST Investigating - Some instances of the admin dash and marketplace are unavailable. We are investigating this issue

Marketplace and admin unavailable

Dec <var data-var='date'>17</var>, <var data-var='time'>11:23</var> EST Completed - Maintenance is complete! Have a good weekend y'all! Dec <var data-var='date'>17</var>, <var data-var='time'>11:20</var> EST Verifying - Verification is currently underway for the maintenance items. Dec <var data-var='date'>17</var>, <var data-var='time'>09:00</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>16</var>, <var data-var='time'>15:50</var> EST Scheduled - We will be doing scheduled upgrades in production between 9am-1pm EST on Saturday 12/17. During this time, scheduled jobs will continue to run, however there may be intermittent slowness in the UI. If this occurs, we expect this to resolve in under 1 minute.

Infrastructure Upgrades

Oct <var data-var='date'>12</var>, <var data-var='time'>13:56</var> EDT Resolved - This incident has been resolved. Oct <var data-var='date'>12</var>, <var data-var='time'>12:20</var> EDT Monitoring - After an earlier incident, we are continuing to monitor as we are experiencing intermittent API slowness.

API Slowness

Oct <var data-var='date'>12</var>, <var data-var='time'>12:10</var> EDT Resolved - This issue is resolved. Oct <var data-var='date'>12</var>, <var data-var='time'>11:47</var> EDT Update - We are continuing to monitor for any further issues. Oct <var data-var='date'>12</var>, <var data-var='time'>11:28</var> EDT Monitoring - All front-ends have recovered. We are continuing to monitor. Oct <var data-var='date'>12</var>, <var data-var='time'>11:19</var> EDT Identified - We have identified the issue and have implemented a fix. APIs are recovering. Oct <var data-var='date'>12</var>, <var data-var='time'>10:55</var> EDT Investigating - We have identified an issue with a core service and are investigating. Scheduled jobs are continuing to run. We will update when we have more information.

Sep <var data-var='date'> 9</var>, <var data-var='time'>10:34</var> EDT Resolved - This issue has been resolved. Apologies for the inconvenience. Sep <var data-var='date'> 9</var>, <var data-var='time'>10:33</var> EDT Monitoring - A fix has been implemented and we are monitoring. Sep <var data-var='date'> 9</var>, <var data-var='time'>10:32</var> EDT Update - We have identified the issue and are deploying a fix. Sep <var data-var='date'> 9</var>, <var data-var='time'>10:28</var> EDT Investigating - The in-app marketplace is unavailable for some customers. We are actively investigating. Runs are continuing but the UI is unavailable.

Is Pandium down?

Pandium status is UP

Lastest published events

API incident due to cloud provider maintenance

API incident in EU region

EU environments experiencing instability

Pandium Services, Components and Regions

Recent Pandium events

API incident due to cloud provider maintenance

API incident in EU region

EU environments experiencing instability

Services inaccessable

Some runs firing with delays

Runs processing with delays