THIS IS A SCHEDULED EVENT Jul <var data-var='date'>12</var>, <var data-var='time'>20:30</var> - <var data-var='time'>23:30</var> PDT Jul <var data-var='date'> 5</var>, <var data-var='time'>22:47</var> PDT Scheduled - We’ll be performing a backend infrastructure upgrade to improve reliability and compliance. During this maintenance window, we do not anticipate any major impact on the Harness platform. We appreciate your understanding and are working to minimize any impact.

Planned Maintenance Notification

THIS IS A SCHEDULED EVENT Jul <var data-var='date'>12</var>, <var data-var='time'>10:00</var> - <var data-var='time'>11:00</var> PDT Jul <var data-var='date'> 2</var>, <var data-var='time'>00:13</var> PDT Scheduled - We’ll be performing a backend infrastructure upgrade to improve reliability and compliance. During this maintenance window, Feature Flags may experience brief, intermittent interruptions to make new changes to the Feature Flag configurations, but the existing flag configurations will continue to be served through out the window. We appreciate your understanding and are working to minimize any impact.

Jul <var data-var='date'> 7</var>, <var data-var='time'>07:10</var> PDT Resolved - This incident has been resolved. Jul <var data-var='date'> 7</var>, <var data-var='time'>06:53</var> PDT Monitoring - We have implemented a fix on our side which has resolved this issue. We are continuing to monitor this issue. Jul <var data-var='date'> 7</var>, <var data-var='time'>03:40</var> PDT Investigating - We are currently investigating an issue in EU1 where users are having issues using IACM module.

IACM module is impacted in EU1

Jul <var data-var='date'> 7</var>, <var data-var='time'>05:36</var> PDT Resolved - This incident has been resolved. Jul <var data-var='date'> 7</var>, <var data-var='time'>04:50</var> PDT Monitoring - A fix has been implemented in one of our microservice responsible for dealing with pipeline templates which has resolved the issue. We are continuing to monitor the issue. Jul <var data-var='date'> 7</var>, <var data-var='time'>03:22</var> PDT Investigating - We are currently investigating an issue in Prod3 cluster where pipeline components are loading slower than expected.

Prod3: Slowness in loading pipeline components

Jul <var data-var='date'> 3</var>, <var data-var='time'>09:11</var> PDT Resolved - This incident has been resolved. Jul <var data-var='date'> 2</var>, <var data-var='time'>16:31</var> PDT Update - We are continuing to monitor results while the backlog is worked through. Jul <var data-var='date'> 2</var>, <var data-var='time'>13:10</var> PDT Update - We are seeing the backlogged items worked through and continuing to monitor results. Jul <var data-var='date'> 2</var>, <var data-var='time'>08:48</var> PDT Monitoring - A fix has been implemented and we are monitoring the results.

FME - Some customers are experiencing delays in scheduled exports of impressions

Jul <var data-var='date'> 1</var>, <var data-var='time'>03:16</var> PDT Completed - The scheduled maintenance has been completed. Jul <var data-var='date'> 1</var>, <var data-var='time'>02:16</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>30</var>, <var data-var='time'>02:18</var> PDT Scheduled - Testing Maintenance

Testing Dev Maintenance

Jun <var data-var='date'>30</var>, <var data-var='time'>13:30</var> PDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>30</var>, <var data-var='time'>09:30</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>26</var>, <var data-var='time'>08:07</var> PDT Update - We will be undergoing scheduled maintenance during this time. Jun <var data-var='date'>22</var>, <var data-var='time'>10:55</var> PDT Scheduled - FME will be undergoing scheduled critical maintenance during this window. No downtime is expected, and all services are anticipated to remain available. As part of this maintenance, updates or modifications to Large Segments will be temporarily unavailable until the activity is completed.

Planned Resiliency activity for Feature Management and Experimentation (FME)

Jun <var data-var='date'>29</var>, <var data-var='time'>08:41</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>29</var>, <var data-var='time'>08:20</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>29</var>, <var data-var='time'>08:20</var> PDT Update - We are continuing to work on a fix for this issue. Jun <var data-var='date'>29</var>, <var data-var='time'>08:02</var> PDT Update - We are continuing to work on a fix for this issue. Jun <var data-var='date'>29</var>, <var data-var='time'>07:57</var> PDT Identified - Harness is investigating an issue affecting our Prod 4 environments. Our log-services are affected and harness is currently working on restoring function.  This can affect our CI, Chaos, IACM modules. Jun <var data-var='date'>29</var>, <var data-var='time'>07:56</var> PDT Investigating - We are currently investigating this issue.

Few pipelines in prod4 are running with degraded performance

Jun <var data-var='date'>28</var>, <var data-var='time'>00:00</var> PDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>27</var>, <var data-var='time'>22:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>10</var>, <var data-var='time'>07:11</var> PDT Scheduled - Our cloud provider is going through a maintenance and during the maintenance window dashboards will not be available to view.

CCM dashboards undergoing maintenance

Jun <var data-var='date'>26</var>, <var data-var='time'>00:31</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>26</var>, <var data-var='time'>00:29</var> PDT Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>26</var>, <var data-var='time'>00:29</var> PDT Monitoring - Issue has been fixed and we are closely monitoring Jun <var data-var='date'>26</var>, <var data-var='time'>00:20</var> PDT Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'>26</var>, <var data-var='time'>00:20</var> PDT Investigating - We are currently experiencing intermittent login issues with the Traceable cluster due to an issue with our login provider. We are actively working with the provider to resolve the issue. This incident does not impact data ingestion, and all ingestion pipelines continue to function normally. We will provide updates as we have more information.

Login to Traceable Clusters Impacted

Jun <var data-var='date'>24</var>, <var data-var='time'>07:37</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>24</var>, <var data-var='time'>05:27</var> PDT Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>24</var>, <var data-var='time'>05:26</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>24</var>, <var data-var='time'>05:03</var> PDT Investigating - Harness login is failing. We are investigating the problem.

Harness Login Failure in EU1 environment

Jun <var data-var='date'>21</var>, <var data-var='time'>21:36</var> PDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>21</var>, <var data-var='time'>20:30</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>19</var>, <var data-var='time'>04:18</var> PDT Scheduled - We’ll be performing a backend infrastructure upgrade to improve reliability and compliance. During this maintenance window, Harness platform may experience brief, intermittent interruptions. We appreciate your understanding and are working to minimize any impact.

Jun <var data-var='date'>17</var>, <var data-var='time'>10:50</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>17</var>, <var data-var='time'>09:47</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>17</var>, <var data-var='time'>09:45</var> PDT Update - IDP Systems' communication was interrupted due to an update.  Harness is rolling back changes. Impacted environments - prod0, prod1, prod2 Jun <var data-var='date'>17</var>, <var data-var='time'>09:45</var> PDT Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'>17</var>, <var data-var='time'>09:44</var> PDT Investigating - We are currently investigating this issue.

IDP Workflows and Backend failures

Jun <var data-var='date'>17</var>, <var data-var='time'>10:13</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>17</var>, <var data-var='time'>09:12</var> PDT Update - We are continuing to monitor for any further issues. Jun <var data-var='date'>17</var>, <var data-var='time'>08:21</var> PDT Update - Maven Central is rate-limiting requests originating from our us-central1 NAT egress IP. Other regions (us-west1, us-east5) are not affected. The Maven/Sonatype team has been engaged and is actively working on lifting the rate limit for our IP range. Jun <var data-var='date'>17</var>, <var data-var='time'>07:21</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>17</var>, <var data-var='time'>07:03</var> PDT Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'>17</var>, <var data-var='time'>06:42</var> PDT Investigating - We are investigating an issue affecting artifact downloads from Maven Central following recent rate-limiting changes implemented by Sonatype. Sonatype has recently implemented rate limits and builds downloading artifacts from central sonatype  https://central.sonatype.org/faq/429-contact-support/. Customers downloading artifacts would be impacted.  We are actively engaging with Sonatype to mitigate the issue here.

Cloud builds are failing with 429 Too Many Requests error while downloading artifacts from central sonatype registry.

Jun <var data-var='date'>14</var>, <var data-var='time'>22:30</var> PDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>14</var>, <var data-var='time'>21:30</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'> 9</var>, <var data-var='time'>04:18</var> PDT Scheduled - We will be performing a backend infrastructure upgrade to improve platform reliability and compliance. During the maintenance window, login access to the Harness platform may experience intermittent interruptions. We are working to minimize any customer impact and appreciate your understanding and patience.

Jun <var data-var='date'> 8</var>, <var data-var='time'>12:50</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'> 8</var>, <var data-var='time'>10:32</var> PDT Monitoring - We are continually monitoring this issue.

Certain customers may see odd web artifacts on Code UI. Logging out and Logging in resolves this issue.

Jun <var data-var='date'> 5</var>, <var data-var='time'>17:30</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'> 5</var>, <var data-var='time'>17:01</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 5</var>, <var data-var='time'>15:56</var> PDT Investigating - We are currently investigating this issue.

Intermittent 503s on IDP services

Jun <var data-var='date'> 4</var>, <var data-var='time'>09:38</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'> 4</var>, <var data-var='time'>08:13</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 4</var>, <var data-var='time'>07:33</var> PDT Identified - Instability in streaming connections could cause a delay in propagating updates. All SDKs are designed to poll against the CDN on streaming disconnects and instability scenarios. This could also cause some noise in the logs as SDKs try to reconnect to streaming. But updates are unaffected.

FME SDKs - SDKs falling back to polling

Jun <var data-var='date'> 1</var>, <var data-var='time'>06:47</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'> 1</var>, <var data-var='time'>04:05</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'> 1</var>, <var data-var='time'>03:01</var> PDT Investigating - We are currently investigating this issue.

Hosted CI Builds Failing Intermittently

May <var data-var='date'>27</var>, <var data-var='time'>18:48</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>27</var>, <var data-var='time'>16:53</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>27</var>, <var data-var='time'>10:54</var> PDT Update - Harness is currently implementing a failover to mitigate the intermittent issue for some customers May <var data-var='date'>27</var>, <var data-var='time'>09:32</var> PDT Update - Harness has implemented a change and are seeing failures reduced, but are continuing to work on completing mitigation. Customers can expect executions to succeed more frequently. Prod1/Prod2 are back to normal. May <var data-var='date'>27</var>, <var data-var='time'>08:20</var> PDT Update - Harness is continuing to investigate and implement changes to fully restore functionality. At this time, we are still seeing some CI Builds intermittently fail. May <var data-var='date'>27</var>, <var data-var='time'>07:06</var> PDT Update - We are continuing to work on a fix for this issue. May <var data-var='date'>27</var>, <var data-var='time'>06:43</var> PDT Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'>27</var>, <var data-var='time'>05:35</var> PDT Investigating - We are currently investigating this issue.

May <var data-var='date'>27</var>, <var data-var='time'>16:20</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>27</var>, <var data-var='time'>07:41</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>27</var>, <var data-var='time'>07:20</var> PDT Investigating - We are currently investigating this issue.

Pipeline logging streams experiencing degraded delivery to console.

May <var data-var='date'>21</var>, <var data-var='time'>09:34</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>14</var>, <var data-var='time'>22:23</var> PDT Update - We are continuing to monitor for any further issues. May <var data-var='date'>14</var>, <var data-var='time'>22:14</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>14</var>, <var data-var='time'>21:54</var> PDT Investigating - We are currently investigating this issue.

IACM Pipeline Struck In Prod4

May <var data-var='date'>21</var>, <var data-var='time'>09:34</var> PDT Resolved - This incident has been resolved. May <var data-var='date'>21</var>, <var data-var='time'>07:49</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>21</var>, <var data-var='time'>07:33</var> PDT Identified - The issue has been identified and a fix is being implemented. May <var data-var='date'>21</var>, <var data-var='time'>07:15</var> PDT Update - Our investigation has revealed that the pipeline execution are happening as expected in the backend, The slowness is limited to updates in pipeline UI only. May <var data-var='date'>21</var>, <var data-var='time'>06:55</var> PDT Investigating - We are currently investigating an slowness issue in pipeline execution ui.

Prod3 pipeline execution UI slowness

May <var data-var='date'>20</var>, <var data-var='time'>08:01</var> PDT Completed - The scheduled maintenance has been completed. May <var data-var='date'> 5</var>, <var data-var='time'>08:01</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'> 4</var>, <var data-var='time'>08:03</var> PDT Scheduled - Action Required: FME SDKs - Certificate pinning changes. We are migrating to a new streaming service for FME, to set ourselves in a good position for future growth and flexibility towards new SDK capabilities. This change will be rolling out incrementally in the coming days. Action required only if you use a combination of: Mobile SDKs with Streaming enabled (default) Certificate pinning, including pinning the streaming.split.io host To avoid delays in rollout updates, please update your SDK certificate pinning configuration as follows: • Add these new trusted hashes for the streaming.split.io hosts to your list: sha256/yxDRqYvQxLYRrI1aLLCiDqU0F3UvNpi1lhL3hs/gUTw= sha256/s1JscMD9xgtY/27NPHzO4zyr6OFXxhr362lmdgcWREY= sha256/rMvxo1yIbyYMwHuas7KtVlwY5UvDs5P44mYfcaa1XUk= sha256/3K7Fs56s8ttWmmzCMddXr11LwV2dRp0Opb2VvZr/mhs= Note: It is important that you keep the prior streaming pin until the rollout is complete. No other changes required. In the event of any issues with connecting the new services, SDKs will fallback to polling. We will notify you once the migration is complete and it is safe to remove the legacy pin. If you have any questions or need assistance, please reach out to our support team.

Action Required: FME SDKs - Certificate pinning changes.

May <var data-var='date'>20</var>, <var data-var='time'>05:38</var> PDT Resolved - This incident has been resolved. The incident was identified as a slowness on the UI updates for execution graph visualization. Executions were not impacted outside of the UI execution graph visualization. May <var data-var='date'>20</var>, <var data-var='time'>04:52</var> PDT Monitoring - A fix has been implemented which resolved the pipeline slowness , we are currently monitoring the results. May <var data-var='date'>20</var>, <var data-var='time'>03:48</var> PDT Investigating - We have received reports of slowness in Harness pipelines in Prod3 cluster. We are actively investigating this issue. Please monitor this space for further updates.

Is Harness down?

Harness status is PARTIAL OUTAGE

Components with issues

Lastest published events

IACM module is impacted in EU1

Prod3: Slowness in loading pipeline components

FME - Some customers are experiencing delays in scheduled exports of impressions

Harness Services, Components and Regions

Recent Harness events

Planned Maintenance Notification

Planned Maintenance Notification

IACM module is impacted in EU1

Prod3: Slowness in loading pipeline components

FME - Some customers are experiencing delays in scheduled exports of impressions

Testing Dev Maintenance

Get your FREE account now,
5 monitors included!

Is Harness down?

Harness status is PARTIAL OUTAGE

Components with issues

Lastest published events

IACM module is impacted in EU1

Prod3: Slowness in loading pipeline components

FME - Some customers are experiencing delays in scheduled exports of impressions

Harness Services, Components and Regions

Recent Harness events

Planned Maintenance Notification

Planned Maintenance Notification

IACM module is impacted in EU1

Prod3: Slowness in loading pipeline components

FME - Some customers are experiencing delays in scheduled exports of impressions

Testing Dev Maintenance

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!