Mar <var data-var='date'>10</var>, <var data-var='time'>21:27</var> CET Resolved - We have identified and resolved the root cause of the issue. One of our nodes was unable to perform DNS resolution due to routing errors. The routing issues have now been fixed, and DNS resolution is working correctly. We apologize for the inconvenience caused.Mar <var data-var='date'>10</var>, <var data-var='time'>20:32</var> CET Update - We identified a DNS resolution issue on one of our production nodes. The Golive service has been moved to another node and is now available again. We are currently working with our cloud provider to determine the root cause of the issue and implement a fix.Mar <var data-var='date'>10</var>, <var data-var='time'>20:04</var> CET Investigating - We are currently investigating the issue.

Outage on Golive

Feb <var data-var='date'> 5</var>, <var data-var='time'>15:56</var> CET Resolved - Over the past hour, we have experienced several traffic peaks on Atlassian webhook calls, and our updated systems have handled the load without any issues. We now consider this issue to be resolved. We apologize for the inconvenience caused.Feb <var data-var='date'> 5</var>, <var data-var='time'>15:01</var> CET Monitoring - A second fix has been deployed to production. Webhook traffic has been re-enabled, and the load appears to be handled correctly. Capabilities such as conflict checking and automation seem to be fully functional. We are continuing to monitor the situation.Feb <var data-var='date'> 5</var>, <var data-var='time'>12:44</var> CET Update - Although the fix helped mitigate the performance issue on one of our components, it is not sufficient to handle the current webhook traffic load. We are continuing to work on performance improvements, but in the meantime, we have had to disable webhook traffic again.Feb <var data-var='date'> 5</var>, <var data-var='time'>12:20</var> CET Identified - A fix has been implemented and deployed to production. We are reopening webhook traffic to assess whether the solution meets performance requirements. This may result in some instability over the next hour.Feb <var data-var='date'> 5</var>, <var data-var='time'>11:18</var> CET Update - Disabling webhook traffic has restored Golive for app navigation. We are currently working on performance fixes so that we can re-enable webhooks (Golive automation and the Scheduling Conflict Checker). We will provide an update soon.Feb <var data-var='date'> 5</var>, <var data-var='time'>09:23</var> CET Update - We have disabled traffic from Atlassian webhooks (issue created/updated/deleted events). As a result, Go-Live automation and scheduling conflict checks are currently unavailable. We are continuing to investigate the issue.Feb <var data-var='date'> 5</var>, <var data-var='time'>09:14</var> CET Update - We are continuing to investigate this issue.Feb <var data-var='date'> 5</var>, <var data-var='time'>08:46</var> CET Investigating - We are currently experiencing a performance issue on Golive that is making the platform inaccessible. We are actively investigating the root cause. We apologize for the inconvenience.

Performance issue on Golive

Dec <var data-var='date'>20</var>, <var data-var='time'>18:43</var> CET Completed - This maintenance must be cancelled as the necessary prerequisites have not been met.Dec <var data-var='date'>15</var>, <var data-var='time'>15:45</var> CET Scheduled - We would like to inform you of a scheduled maintenance related to the transfer of our domain names for the Golive and Time Squad Cloud applications. This operation is not expected to cause any service interruption. However, during the maintenance window, minor and temporary access delays may occur. Thank you for your understanding.

Domain name transfer

Nov <var data-var='date'> 5</var>, <var data-var='time'>11:07</var> CET Resolved - This incident has been resolved.Nov <var data-var='date'> 5</var>, <var data-var='time'>09:19</var> CET Monitoring - Our hosting provider has resolved the issue. You can find details about the incident on their status page: https://public-cloud.status-ovhcloud.com/incidents/9qmxv0gbky43 All systems are currently being monitored, and both Golive and Time Squad are now fully operational. Thank you for your patience and understanding throughout this incident.Nov <var data-var='date'> 5</var>, <var data-var='time'>07:58</var> CET Update - We’ve identified that the issue is coming from our hosting provider. We’re currently in contact with their teams and are working together to resolve the problem as soon as possible. Thank you for your continued patience and understanding.Nov <var data-var='date'> 5</var>, <var data-var='time'>07:20</var> CET Investigating - We are currently experiencing an outage affecting Golive and Time Squad since 6:00 AM UTC. Our team is actively working to resolve the issue as quickly as possible. We apologize for the inconvenience and appreciate your patience.

Golive and Time Squad outage

Oct <var data-var='date'>25</var>, <var data-var='time'>20:00</var> CEST Completed - The scheduled maintenance has been completed.Oct <var data-var='date'>25</var>, <var data-var='time'>19:00</var> CEST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Oct <var data-var='date'>17</var>, <var data-var='time'>07:46</var> CEST Scheduled - Upgrade of kubernetes. During this period, users should expect some instabilities of Golive & Timesquad.

Infrastructure upgrade

Oct <var data-var='date'>23</var>, <var data-var='time'>13:29</var> CEST Completed - Maintenance has been completed. DNS propagation is now complete, and the old load balancers have been decommissioned.Oct <var data-var='date'>22</var>, <var data-var='time'>13:29</var> CEST Update - We have completed the DNS switch. During the transition, a brief outage lasting from a few seconds up to 1–2 minutes may have occurred due to a missing configuration on our Web Application Firewall. All traffic should now be routed through the new IP address: 141.94.164.105 To verify that your local DNS record has been updated, you can run: dig golive.apwide.net or, for Time Squad: dig timeline.apwide.net You should see an answer showing the IP address 141.94.164.105. We will continue serving traffic on both IP addresses for the next 24 hours to ensure full DNS propagation.Oct <var data-var='date'>22</var>, <var data-var='time'>13:00</var> CEST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Oct <var data-var='date'>15</var>, <var data-var='time'>14:36</var> CEST Scheduled - Due to a deprecation notice from our hosting provider, we are currently migrating the load balancers (LB) that handle traffic for Golive and Time Squad. If you have whitelisted our IP addresses, we kindly ask you to add the new load balancer IP address 141.94.164.105 to your existing firewall rules. The new load balancer is already operational and will run in parallel with the old one during the upcoming week. You can verify that your firewall allows access by sending a request to the new IP with the appropriate host header — you should receive an HTTP 308 response. Example: curl -v http://141.94.164.105 -H "Host: golive.apwide.net" For comparison, you can test the old load balancer as follows: curl -v http://51.178.69.84 -H "Host: golive.apwide.net" Once the maintenance begins, we will perform a DNS switch to route traffic for Golive and Time Squad to the new IP address. DNS propagation should complete globally within 24 hours, after which all traffic will be served through the new load balancer. During this migration window, if your firewall rules have not been updated, you may experience connection issues or be temporarily unable to access Golive/Time Squad. Therefore, we strongly recommend contacting your network team as soon as possible to ensure a smooth transition. Thank you for your understanding and cooperation.

Load balancer migration

Apr <var data-var='date'>29</var>, <var data-var='time'>10:00</var> CEST Resolved - Golive & Time Squad were unavailable this morning for a duration of approximately 30 minutes. We are currently conducting an investigation to identify the root cause.

Unexpected outage

Apr <var data-var='date'>17</var>, <var data-var='time'>03:30</var> CEST Resolved - Golive & Time Squad were unavailable this morning for a duration of 1 hour and 10 minutes. The root cause of the issue was a problem accessing the application databases. After contacting our hosting provider, they confirmed that a network maintenance was scheduled for this morning. Traffic was supposed to be rerouted, but a configuration issue caused a routing error. We apologize for the inconvenience caused.

Unexpected outage on managed database

Jan <var data-var='date'>18</var>, <var data-var='time'>19:46</var> CET Completed - The scheduled maintenance has been completed.Jan <var data-var='date'>18</var>, <var data-var='time'>19:00</var> CET In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Dec <var data-var='date'>17</var>, <var data-var='time'>08:19</var> CET Scheduled - Upgrade of kubernetes. During this period, users should expect some instabilities of Golive & Timesquad.

Jul <var data-var='date'> 8</var>, <var data-var='time'>12:38</var> CEST Resolved - Infrastructure has been patched and issue referenced by https://public-cloud.status-ovhcloud.com/incidents/ffhr44srdcln is now fixed on our cluster. We apologize for the inconvenience it caused.Jul <var data-var='date'> 8</var>, <var data-var='time'>12:17</var> CEST Update - Here is the reference to the issue currently impacting our cluster: https://public-cloud.status-ovhcloud.com/incidents/ffhr44srdclnJul <var data-var='date'> 8</var>, <var data-var='time'>12:13</var> CEST Investigating - Following our upgrade/outage of 2024-07-08, our hosting provider has to exceptionally patch our nodes. Golive/Time Squad won't be available for 30 minutes. Sorry for this inconvenience.

Infrastructure being patched

Jul <var data-var='date'> 7</var>, <var data-var='time'>12:30</var> CEST Completed - The scheduled maintenance has been completed.Jul <var data-var='date'> 7</var>, <var data-var='time'>11:59</var> CEST Update - Our infrastructure upgrade takes longer than expected, we expect the maintenance to take 30 minutes more than initially planned.Jul <var data-var='date'> 7</var>, <var data-var='time'>10:00</var> CEST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Jun <var data-var='date'>26</var>, <var data-var='time'>17:09</var> CEST Scheduled - We will proceed with a release upgrade that requires an outage due to a specific database upgrade. We will take advantage of this downtime to also upgrade our Kubernetes version. During this period, users will not be able to access Golive and Time Squad.

Release rollout + Infrastructure upgrade

Feb <var data-var='date'>29</var>, <var data-var='time'>16:09</var> CET Resolved - Issue has been fixed on hosting provider side. Issue was due to a "CPU spikes on the ingress gateway" (load balancers exposing Golive & TIme Squad application on the web). We apologize for any inconvenience Thank you for your patience.Feb <var data-var='date'>29</var>, <var data-var='time'>16:02</var> CET Investigating - We are currently experiencing an issue with our hosting provider, causing our applications to be temporarily unavailable. We apologize for any inconvenience and are actively working to resolve the issue as quickly as possible. Thank you for your patience.

There is an ongoing issue with our hosting provider, which is causing our applications to be unavailable

Feb <var data-var='date'> 3</var>, <var data-var='time'>21:13</var> CET Completed - The scheduled maintenance has been completed.Feb <var data-var='date'> 3</var>, <var data-var='time'>20:00</var> CET In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Jan <var data-var='date'>16</var>, <var data-var='time'>10:18</var> CET Scheduled - Upgrade of kubernetes. During this period, users should expect some instabilities of Golive & Timesquad.

Infrastructure Upgrade

Feb <var data-var='date'> 1</var>, <var data-var='time'>14:03</var> CET Resolved - The issue appears to be successfully addressed. It stemmed from contention within our system responsible for communication with Jira instances. To remedy this, we have refined the initiation of calls to Jira, reducing their frequency. Additionally, we have augmented the pool size of the system overseeing the contention, enhancing its capacity to handle tasks effectively.Feb <var data-var='date'> 1</var>, <var data-var='time'>10:09</var> CET Monitoring - A fix has been implemented and we are monitoring the results.Jan <var data-var='date'>31</var>, <var data-var='time'>08:29</var> CET Identified - A potential root cause has been pinpointed, and a configuration workaround has been implemented in the production environment. We are actively monitoring its impact. Concurrently, efforts are underway to develop a permanent fix.Jan <var data-var='date'>30</var>, <var data-var='time'>22:51</var> CET Update - Platform is stable for 5 hours but analysis to identify the root cause is still ongoing.Jan <var data-var='date'>30</var>, <var data-var='time'>16:24</var> CET Investigating - Apwide Golive is currently experiencing instabilities. Our teams are working to identify the causes.

Golive instabilities

Dec <var data-var='date'>11</var>, <var data-var='time'>04:30</var> CET Resolved - During a short period of time (3:52am UTC to 3:56am UTC), Golive tomcat thread pool was saturated by a high number of requests and was not available. In details: 3:22am UTC: Tomcat thread pool consumption slowly increased (usually 10 threads out of 200 available but increased to 18) but was still able to handle the load. 3:39am UTC: Thread pool consumption increase accelerated. 3:52am UTC: No more available thread in the pool (200 busy threads), Tomcat stopped handling new requests. (outage) 3:56am UTC: Current busy threads completed their tasks, most of them are released, and processing was back to normal. (10 threads out of 200) We are sorry for the inconvenience it caused and we are currently working on the app to mitigate the risk of this happening again.

Golive unavailable

Nov <var data-var='date'>16</var>, <var data-var='time'>16:40</var> CET Resolved - This incident has been resolvedNov <var data-var='date'>16</var>, <var data-var='time'>14:40</var> CET Monitoring - A fix has been implemented by our hosting provider and the service is restored. We are monitoring the situation.Nov <var data-var='date'>16</var>, <var data-var='time'>12:08</var> CET Update - Our hosting provider is still working on a fix and we are actively following up. There is no estimated resolution time for the time being.Nov <var data-var='date'>16</var>, <var data-var='time'>08:45</var> CET Update - A fix is being implemented by our hosting provider.Nov <var data-var='date'>16</var>, <var data-var='time'>02:55</var> CET Identified - The issue has been identified by our hosting provider and a fix is being implemented. More information: https://public-cloud.status-ovhcloud.com/incidents/6sd9lwym7zdtNov <var data-var='date'>16</var>, <var data-var='time'>02:50</var> CET Investigating - We are currently investigating this issue.

Major outage

Aug <var data-var='date'>24</var>, <var data-var='time'>13:07</var> CEST Resolved - New patch applied on production has fixed the issue and monitoring does not show any new occurrence.Aug <var data-var='date'>24</var>, <var data-var='time'>10:56</var> CEST Update - Fixed deployed to production has only partially solved the problem. Currently, search endpoint is not able to select correct output format in case client calling the API does not specify "accept" content type. A new fix is in progress.Aug <var data-var='date'>24</var>, <var data-var='time'>09:50</var> CEST Monitoring - A fix has been pushed to production and we are currently monitoring its impact.Aug <var data-var='date'>24</var>, <var data-var='time'>08:29</var> CEST Investigating - Some calls to Rest API endpoint GET /environments/search/paginated ends with a HTTP 405. This can impact an integration made with Rest API including Jenkins Shared Library. A fix is in progress.

Issue on Rest API

Jun <var data-var='date'> 3</var>, <var data-var='time'>19:00</var> CEST Completed - The scheduled maintenance has been completed.Jun <var data-var='date'> 3</var>, <var data-var='time'>18:01</var> CEST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.May <var data-var='date'>30</var>, <var data-var='time'>07:45</var> CEST Scheduled - Upgrade of kubernetes. During this period, users should expect some instabilities of Golive & Timesquad.

Mar <var data-var='date'>28</var>, <var data-var='time'>01:51</var> CEST Resolved - This incident has been resolved.Mar <var data-var='date'>28</var>, <var data-var='time'>01:51</var> CEST Update - After a configuration change, issue seems to be definitely resolved.Mar <var data-var='date'>28</var>, <var data-var='time'>01:11</var> CEST Monitoring - A fixed has been applied on production and seems to have fixed the issue. Environments are under monitoringMar <var data-var='date'>28</var>, <var data-var='time'>00:00</var> CEST Investigating - We are currently investigating this issue.

Mar <var data-var='date'>24</var>, <var data-var='time'>10:20</var> CET Resolved - Incident is now resolved. Issue was due to the upgrade of Jira cloud rest API client to the v3 which resulted in some email addresses not correctly retrieved by email sending job.Mar <var data-var='date'>23</var>, <var data-var='time'>20:07</var> CET Update - We are continuing to monitor for any further issues.Mar <var data-var='date'>23</var>, <var data-var='time'>20:06</var> CET Monitoring - Fix has been pushed to production and we're currently monitoring if issue is solved.Mar <var data-var='date'>23</var>, <var data-var='time'>18:47</var> CET Identified - We've identified the potential root cause of this issue. A fix is in progress.Mar <var data-var='date'>23</var>, <var data-var='time'>17:57</var> CET Investigating - We're currently investigating possible issues impacting emails sent by Golive automation engine and watcher capability.

Some email notifications are not sent correctly

Jan <var data-var='date'>17</var>, <var data-var='time'>03:00</var> CET Resolved - Following an operational/infrastructure change (update of our API gateway), Golive Rest API was in partial outage. Impact: Golive frontend was up and running, but some generated Rest API tokens were considered revoked. This resulted in some failing API calls (HTTP 500). Actions (UTC time): - 3:00am: rollback operational change - 6:32am: re-apply change with first attempt of resolution + monitoring - 8:00am: new occurrences identified on production - 9:32am: push fix for the second attempt of resolution - 10:30am: still occurrences of error found on production - 1:10pm: apply new fix on API gateway + monitoring - 4pm: no more occurrences, problem seems fixed We apologize for the inconvenience caused.

Partial outage on Golive Rest API

Nov <var data-var='date'>20</var>, <var data-var='time'>16:30</var> CET Completed - The scheduled maintenance has been completed.Nov <var data-var='date'>20</var>, <var data-var='time'>10:00</var> CET In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Nov <var data-var='date'>18</var>, <var data-var='time'>09:59</var> CET Update - We will be undergoing scheduled maintenance during this time.Nov <var data-var='date'>14</var>, <var data-var='time'>15:14</var> CET Scheduled - During this period, Golive & Time Squad cloud won't be available.

Infrastructure & Apps Upgrades

Oct <var data-var='date'>24</var>, <var data-var='time'>21:00</var> CEST Resolved - This incident has been resolved.Oct <var data-var='date'>24</var>, <var data-var='time'>19:53</var> CEST Monitoring - The service is back and we are monitoring it closely.Oct <var data-var='date'>24</var>, <var data-var='time'>19:21</var> CEST Investigating - We are currently investigating this issue.

Sep <var data-var='date'>20</var>, <var data-var='time'>11:40</var> CEST Resolved - Golive and Time Squad cloud were not available during 5 minutes due a network issue to our hosting provider. More details here: https://network.status-ovhcloud.com/incidents/5mldyhd6v99c

Network degradation

Jul <var data-var='date'>23</var>, <var data-var='time'>21:26</var> CEST Completed - The scheduled maintenance has been completed.Jul <var data-var='date'>23</var>, <var data-var='time'>20:15</var> CEST Verifying - Verification is currently underway for the maintenance items.Jul <var data-var='date'>23</var>, <var data-var='time'>20:14</var> CEST Update - Scheduled maintenance is still in progress. We will provide updates as necessary.Jul <var data-var='date'>23</var>, <var data-var='time'>20:00</var> CEST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Jul <var data-var='date'>20</var>, <var data-var='time'>16:36</var> CEST Scheduled - Our cloud applications won't be available during this timeframe. Maintenance scope: - upgrade our infrastructure orchestrator. - proceed to a database cleaning of Golive webhook notifications table.

Is Apwide down?

Apwide status is UP

Lastest published events

Outage on Golive

Performance issue on Golive

Domain name transfer

Apwide Services, Components and Regions

Recent Apwide events

Outage on Golive

Performance issue on Golive

Domain name transfer

Golive and Time Squad outage

Infrastructure upgrade

Load balancer migration

Get your FREE account now,
5 monitors included!

Is Apwide down?

Apwide status is UP

Lastest published events

Outage on Golive

Performance issue on Golive

Domain name transfer

Apwide Services, Components and Regions

Recent Apwide events

Outage on Golive

Performance issue on Golive

Domain name transfer

Golive and Time Squad outage

Infrastructure upgrade

Load balancer migration

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!