Oct <var data-var='date'>13</var>, <var data-var='time'>06:00</var> EDT Completed - The scheduled maintenance has been completed. Oct <var data-var='date'>13</var>, <var data-var='time'>05:47</var> EDT Update - Extending the window until 6:30am eastern. Oct <var data-var='date'>13</var>, <var data-var='time'>05:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'> 9</var>, <var data-var='time'>19:11</var> EDT Scheduled - You should expect between 2 and 6 minutes of downtime for our APIs, classic Merchant App and new Merchant App between 5 and 6am eastern on Oct 13.

Planned downtime (2-6 minutes) to complete database major version upgrade

Oct <var data-var='date'> 4</var>, <var data-var='time'>19:05</var> EDT Resolved - Background job processing went down on 10/1 or us-east-1 region and 10/4 for eu-west-1 region. Both incidents were due to the same root cause: * A key scheduling service failed. * This was triggered by our core transactional database recycling after a minor version upgrade. * This happened around 3am eastern time on 10/4 in us-east-1 region. * Happened around 3am eastern time on 10/1 in eu-west-1 region. * We should have noticed this, however there was an error in some of our alerting / notification scripts. * We have since updated and thoroughly tested the notification scripts.

Background job processing (including ordering jobs) down

Apr <var data-var='date'>16</var>, <var data-var='time'>13:42</var> EDT Resolved - The issue has been resolved. Apr <var data-var='date'>16</var>, <var data-var='time'>13:26</var> EDT Investigating - We are currently investigating an issue causing an outage of our Merchant App and API.

Merchant App and API Outage

Feb <var data-var='date'> 9</var>, <var data-var='time'>14:05</var> EST Resolved - We have identified the core issue and a fix is in place. We will follow up with a postmortem. Feb <var data-var='date'> 9</var>, <var data-var='time'>11:59</var> EST Update - All systems are currently operating normally. We are still monitoring. We will post a postmortem write up once we fully understand the root cause. Feb <var data-var='date'> 9</var>, <var data-var='time'>11:59</var> EST Monitoring - We have implemented a fix and are monitoring. Please contact support@subscribepro.com if you continue to experience performance or connectivity issues. Feb <var data-var='date'> 9</var>, <var data-var='time'>10:47</var> EST Investigating - We are currently investigating an issue causing degraded performance of our Merchant App and API.

Degraded performance of Merchant App and API

Feb <var data-var='date'> 8</var>, <var data-var='time'>08:53</var> EST Resolved - This incident has been reported resolved by our upstream provider, and we are also no longer seeing any issues, so we are also considering it resolved. Feb <var data-var='date'> 8</var>, <var data-var='time'>08:32</var> EST Monitoring - Our upstream provider has implemented a fix and transactions appear to be processing successfully now. We are actively monitoring the results. Feb <var data-var='date'> 8</var>, <var data-var='time'>08:25</var> EST Identified - Currently all payment transactions submitted through the Subscribe Pro Vault are failing. All other APIs and systems are operational. We have identified an issue with an upstream provider causing failing transaction requests. They have published an incident here: https://status.spreedly.com/incidents/1ly5j04m8mjs

All payment transactions failing

Jan <var data-var='date'>22</var>, <var data-var='time'>06:17</var> EST Completed - Maintenance is completed. No downtime was required. Jan <var data-var='date'>22</var>, <var data-var='time'>05:35</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>22</var>, <var data-var='time'>05:34</var> EST Scheduled - We will be upgrading some underlying hardware as part of a deployment. This will possibly require 1-2 minutes of downtime during this period.

Upgrade in progress

Oct <var data-var='date'> 1</var>, <var data-var='time'>06:40</var> EDT Resolved - Full service has been restored. Oct <var data-var='date'> 1</var>, <var data-var='time'>06:17</var> EDT Investigating - Our Merchant App and API are currently experiencing partial downtime due to a long running database update.

Currently experiencing partial downtime

Sep <var data-var='date'>17</var>, <var data-var='time'>16:26</var> EDT Resolved - This incident has been resolved. Sep <var data-var='date'>17</var>, <var data-var='time'>16:13</var> EDT Monitoring - A fix has been completed and performance has returned to normal. We will continue to monitor. Sep <var data-var='date'>17</var>, <var data-var='time'>16:03</var> EDT Investigating - We are currently investigating degraded performance of our Merchant App and API.

Degraded Performance of our Merchant App and API.

Aug <var data-var='date'>11</var>, <var data-var='time'>22:23</var> EDT Resolved - The issue causing a partial outage of our Merchant App and API is resolved. Aug <var data-var='date'>11</var>, <var data-var='time'>22:13</var> EDT Investigating - We are currently investigating a partial outage of our Merchant App and API.

Partial Outage

Jul <var data-var='date'>24</var>, <var data-var='time'>08:00</var> EDT Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>24</var>, <var data-var='time'>06:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>20</var>, <var data-var='time'>15:38</var> EDT Scheduled - Subscribe Pro will be performing scheduled changes on the morning of Friday, July 24, 2020, between 6am and 7am EDT. We are making improvements to our infrastructure to increase capacity and ensure additional scalability as we grow. We do not expect any downtime during these changes. As a result of these updates, the IP addresses used to connect with your eCommerce platform and send webhook notifications are changing. If you require to whitelist Subscribe Pro IP addresses for access, please Whitelist the Following IPs before July 24, 2020: 18.235.153.246 18.235.22.0 23.20.232.129 For more information, you can see our published list of IP addresses (https://docs.subscribepro.com/technical/ip-addresses/). If you have any additional questions, please contact support@subscribepro.com.

Upcoming Subscribe Pro IP Address and Infrastructure Change

Jul <var data-var='date'>17</var>, <var data-var='time'>20:39</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'>17</var>, <var data-var='time'>17:41</var> EDT Monitoring - The issue with our upstream provider, Cloudflare, appears to be resolved. We are continuing to monitor. Jul <var data-var='date'>17</var>, <var data-var='time'>17:29</var> EDT Update - We are continuing to investigate this issue. Jul <var data-var='date'>17</var>, <var data-var='time'>17:29</var> EDT Investigating - We are currently investigating an outage of our Merchant App UI and API. Currently it looks like an issue with our upstream provider, CloudFlare. We will keep you updated as we learn more.

Merchant App UI and API Outage

Jun <var data-var='date'>29</var>, <var data-var='time'>15:06</var> EDT Resolved - We identified an issue during our most recent deployment that resulted in traffic to our API and web UI not being processed. The relevant change was reverted and the issue has been resolved. Jun <var data-var='date'>29</var>, <var data-var='time'>14:40</var> EDT Investigating - We are currently identifying an issue with our merchant and and API.

Merchant app and API Outage

May <var data-var='date'>30</var>, <var data-var='time'>09:11</var> EDT Resolved - A fix has been deployed by Spreedly and the core issue has been resolved. Transactions are processing successfully. May <var data-var='date'>30</var>, <var data-var='time'>08:31</var> EDT Monitoring - We have implemented a stable workaround. The core issue is related to an SSL certificate on core.spreedly.com. We will update when the core issue has been resolved. May <var data-var='date'>30</var>, <var data-var='time'>07:25</var> EDT Investigating - We are currently investigating an issue causing payment transactions to fail.

Transaction Processing Issues

May <var data-var='date'>26</var>, <var data-var='time'>06:35</var> EDT Resolved - The issue has been resolved by our upstream provider. May <var data-var='date'>26</var>, <var data-var='time'>06:13</var> EDT Monitoring - A fix has been applied by our upstream provider. Services are currently up and being monitored. May <var data-var='date'>26</var>, <var data-var='time'>05:39</var> EDT Investigating - This incident is due to issues at our upstream provider. We will keep you updated as it is investigated and resolved. https://status.platform.sh/

Outage: API and Merchant App

Apr <var data-var='date'>24</var>, <var data-var='time'>08:19</var> EDT Resolved - Network issue was marked resolved by Cloudflare at 4:25am eastern. Apr <var data-var='date'>23</var>, <var data-var='time'>23:05</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Apr <var data-var='date'>23</var>, <var data-var='time'>23:04</var> EDT Investigating - We experienced a brief period of downtime (3-5 minutes) starting around 10:49pm eastern. The issue is related to a Cloudflare outage in Ashburn, VA. https://www.cloudflarestatus.com/ At this point, service has been restored and we are monitoring the issue.

Partial Network Outage

Mar <var data-var='date'> 1</var>, <var data-var='time'>07:30</var> EST Completed - The scheduled maintenance has been completed. Mar <var data-var='date'> 1</var>, <var data-var='time'>07:10</var> EST Update - Database maintenance completed. Mar <var data-var='date'> 1</var>, <var data-var='time'>05:30</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'>21</var>, <var data-var='time'>09:35</var> EST Scheduled - We will be performing minor maintenance on the Subscribe Pro platform primary databases. We will begin at 5:30am eastern time on March 1. We expect our systems to be down for somewhere between 1 and 5 minutes during the 1 hour maintenance window we are scheduling. We will be adjusting some database configuration and updating SSL certificates on the database.

Minor Database Maintenance

Sep <var data-var='date'>24</var>, <var data-var='time'>13:10</var> EDT Resolved - This incident has been resolved. Sep <var data-var='date'>24</var>, <var data-var='time'>12:00</var> EDT Monitoring - A fix is in place and we are monitoring. Sep <var data-var='date'>24</var>, <var data-var='time'>11:51</var> EDT Investigating - We are investigating a partial outage of our Merchant UI and API

Partial Outage of the Merchant UI and API

Jul <var data-var='date'>30</var>, <var data-var='time'>14:53</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'>30</var>, <var data-var='time'>14:14</var> EDT Monitoring - We have fixed the issue causing some recurring orders not to be processed. We will continue to monitor for any further issues. Jul <var data-var='date'>30</var>, <var data-var='time'>14:08</var> EDT Investigating - We are investigating reports that some recurring orders are not being processed as expected.

Recurring Orders not Processing

Jul <var data-var='date'>30</var>, <var data-var='time'>14:03</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'>30</var>, <var data-var='time'>13:32</var> EDT Monitoring - A fix has been implemented and we are monitoring. If you continue to experience issues accessing Merchant UI functionality, please contact support@subscribepro.com Jul <var data-var='date'>30</var>, <var data-var='time'>08:56</var> EDT Investigating - We are currently investigating a 500 Internal Server Error that is preventing access to subscription and customer management pages in our Merchant UI.

Internal Server Error on Subscription and Customer Management pages in Merchant UI

Jul <var data-var='date'>19</var>, <var data-var='time'>09:24</var> EDT Resolved - This outage has been resolved. Please contact support@subscribepro.com if you experience any further issues accessing our platform web UI or API. Jul <var data-var='date'>19</var>, <var data-var='time'>07:20</var> EDT Monitoring - A fix has been implemented and we are monitoring. Jul <var data-var='date'>19</var>, <var data-var='time'>06:50</var> EDT Investigating - We are currently investigating an outage of our platform UI and API.

Platform UI and API Outage

Jul <var data-var='date'> 2</var>, <var data-var='time'>11:01</var> EDT Resolved - Cloudflare has considered this issue resolved. Affected services have resumed normal operation. You can find more information about this issue here: https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr Jul <var data-var='date'> 2</var>, <var data-var='time'>10:19</var> EDT Monitoring - Cloudflare has implemented a fix. Service appears to be restored. We will monitor and provide an update when the issue is resolved. Jul <var data-var='date'> 2</var>, <var data-var='time'>09:53</var> EDT Investigating - Issue was determined to be caused by Cloudflare outage - https://www.cloudflarestatus.com/

Hosted Widgets

Jun <var data-var='date'>24</var>, <var data-var='time'>09:51</var> EDT Resolved - The root cause was a Cloudflare outage. Cloudflare service has been fully restored. Subscribe Pro hosted code is now fully operational as well. https://www.cloudflarestatus.com/ Jun <var data-var='date'>24</var>, <var data-var='time'>09:49</var> EDT Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>24</var>, <var data-var='time'>07:06</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>24</var>, <var data-var='time'>07:06</var> EDT Update - Issue was determined to be caused by Cloudflare outage - https://www.cloudflarestatus.com/ We have temporarily bypassed the issue and hosted code should be available. Jun <var data-var='date'>24</var>, <var data-var='time'>06:55</var> EDT Investigating - We're currently investigating a partial outage of our hosted widget code. The cause seems to be related to a Cloudflare connection issue.

Apr <var data-var='date'>14</var>, <var data-var='time'>11:46</var> EDT Resolved - The partial outage of recurring order processing has been resolved. Apr <var data-var='date'>14</var>, <var data-var='time'>11:08</var> EDT Monitoring - We have identified the issue and put a fix into place. We will continue to monitor to ensure everything continues to process as expected. If your accounts' subscriptions were delayed, they should process soon. Apr <var data-var='date'>14</var>, <var data-var='time'>10:41</var> EDT Investigating - We are currently investigating a partial outage of our recurring order processing.

Jan <var data-var='date'>31</var>, <var data-var='time'>10:08</var> EST Resolved - We experience brief intermittent downtime due to an issue with our logging cluster and application logging queue. Full service has been restored. Jan <var data-var='date'>31</var>, <var data-var='time'>09:58</var> EST Investigating - We are currently investigating a partial outage of our platform UI and API.

Nov <var data-var='date'> 8</var>, <var data-var='time'>00:38</var> EST Resolved - API and UI are operating normally. Nov <var data-var='date'> 7</var>, <var data-var='time'>23:47</var> EST Investigating - We are currently investigating intermittent 502, 503 and 504 errors.

Is Subscribe Pro down?

Subscribe Pro status is UP

Lastest published events

Planned downtime (2-6 minutes) to complete database major version upgrade

Background job processing (including ordering jobs) down

Merchant App and API Outage

Subscribe Pro Services, Components and Regions

Recent Subscribe Pro events

Planned downtime (2-6 minutes) to complete database major version upgrade

Background job processing (including ordering jobs) down

Merchant App and API Outage

Degraded performance of Merchant App and API

All payment transactions failing

Upgrade in progress