May <var data-var='date'>20</var>, <var data-var='time'>15:32</var> UTC Resolved - This incident has been resolved. May <var data-var='date'>20</var>, <var data-var='time'>15:02</var> UTC Monitoring - A fix has been implemented, and everything should be back to normal. The previously shared snippets can be removed. May <var data-var='date'>20</var>, <var data-var='time'>13:59</var> UTC Identified - We identified slow GitHub operations related to problems with the network, and we deployed a fix to re-route those operations to a healthier server. Some HTTPS operations started to fail because of that. To disable the fix, you should use the command: sudo sed -i 's/^\([0-9].*github\.com\)/#\1/' /etc/hosts And to re-enable, you can use: sudo sed -i 's/^#\([0-9].*github\.com\)/\1/' /etc/hosts May <var data-var='date'>20</var>, <var data-var='time'>12:48</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>20</var>, <var data-var='time'>08:44</var> UTC Identified - We identified slow communication between Semaphore jobs and GitHub. We are continuing to work with our upstream provider to resolve this. For impacted jobs, please replace the normal checkout command with the following: retry --times 5 "timeout 40 bash -lc 'source ~/.toolbox/toolbox; checkout'" And afterwards, use cd to change the directory to the cloned repository (replace with your project).

Delays when checking out GitHub repos via SSH

May <var data-var='date'>19</var>, <var data-var='time'>15:42</var> UTC Resolved - This incident has been resolved. May <var data-var='date'>19</var>, <var data-var='time'>15:10</var> UTC Monitoring - We are not currently seeing timeouts related to slow connections. Our team is continuing to monitor the situation to confirm that service has fully returned to normal. May <var data-var='date'>19</var>, <var data-var='time'>11:18</var> UTC Identified - We identified that the issue is slow communication between our jobs and GitHub. We are currently working with our upstream provider to to resolve this. For impacted jobs, please replace the normal checkout command with the following: retry --times 5 "timeout 40 bash -lc 'source ~/.toolbox/toolbox; checkout'" And afterwards, use cd <dir> to change the directory to the cloned repository (replace <dir> with your project). May <var data-var='date'>19</var>, <var data-var='time'>10:38</var> UTC Update - We are continuing to investigate this issue. May <var data-var='date'>19</var>, <var data-var='time'>10:00</var> UTC Investigating - We are currently investigating this issue.

Mar <var data-var='date'>12</var>, <var data-var='time'>17:45</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>12</var>, <var data-var='time'>17:11</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'>12</var>, <var data-var='time'>17:03</var> UTC Identified - The issue has been identified and a fix is being implemented. Mar <var data-var='date'>12</var>, <var data-var='time'>15:55</var> UTC Investigating - We are currently investigating this issue.

Delays in job processing

Mar <var data-var='date'>11</var>, <var data-var='time'>17:42</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>11</var>, <var data-var='time'>17:18</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'>11</var>, <var data-var='time'>16:28</var> UTC Identified - The issue has been identified and a fix is being implemented.

Mar <var data-var='date'> 8</var>, <var data-var='time'>18:52</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 8</var>, <var data-var='time'>18:22</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'> 8</var>, <var data-var='time'>17:26</var> UTC Identified - The issue has been identified and a fix is being implemented. Mar <var data-var='date'> 8</var>, <var data-var='time'>16:25</var> UTC Update - We are continuing to investigate this issue. Mar <var data-var='date'> 8</var>, <var data-var='time'>15:21</var> UTC Investigating - We are currently investigating this issue.

Mar <var data-var='date'> 3</var>, <var data-var='time'>18:05</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 3</var>, <var data-var='time'>17:36</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'> 3</var>, <var data-var='time'>17:06</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'>24</var>, <var data-var='time'>10:32</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>24</var>, <var data-var='time'>10:05</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. We will provide our next update within 30 minutes or sooner if things change. Feb <var data-var='date'>24</var>, <var data-var='time'>09:54</var> UTC Identified - The issue has been identified and we are working on a fix. We will provide our next update within 30 minutes or sooner if things change. We thank you for your patience while we continue to work to resolve this issue. Feb <var data-var='date'>24</var>, <var data-var='time'>09:34</var> UTC Update - We are continuing to investigate job start failures and processing delays. We will continue to update while investigating this. Feb <var data-var='date'>24</var>, <var data-var='time'>09:06</var> UTC Update - We are continuing to investigate this issue. Feb <var data-var='date'>24</var>, <var data-var='time'>08:49</var> UTC Investigating - We are currently investigating this issue.

Jobs Not Starting and Processing Delays

Feb <var data-var='date'>20</var>, <var data-var='time'>18:43</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>20</var>, <var data-var='time'>18:17</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>20</var>, <var data-var='time'>17:30</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>20</var>, <var data-var='time'>15:56</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'>18</var>, <var data-var='time'>20:41</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>18</var>, <var data-var='time'>20:12</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>18</var>, <var data-var='time'>19:12</var> UTC Update - Our team is still working on a fix for this issue. Feb <var data-var='date'>18</var>, <var data-var='time'>18:18</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>18</var>, <var data-var='time'>17:07</var> UTC Identified - The issue has been identified and a fix is being implemented. Feb <var data-var='date'>18</var>, <var data-var='time'>16:13</var> UTC Investigating - We are currently investigating this issue.

Feb <var data-var='date'>17</var>, <var data-var='time'>18:18</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>17</var>, <var data-var='time'>17:50</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>17</var>, <var data-var='time'>17:45</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>17</var>, <var data-var='time'>16:34</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>17</var>, <var data-var='time'>16:32</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'> 7</var>, <var data-var='time'>11:00</var> UTC Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 7</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'> 3</var>, <var data-var='time'>13:56</var> UTC Scheduled - Semaphore will be temporarily unavailable during this scheduled maintenance on Saturday.

Infrastructure maintenance

Feb <var data-var='date'> 5</var>, <var data-var='time'>17:28</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 5</var>, <var data-var='time'>16:28</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'> 4</var>, <var data-var='time'>18:10</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 4</var>, <var data-var='time'>17:40</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'> 4</var>, <var data-var='time'>16:38</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'> 3</var>, <var data-var='time'>19:50</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 3</var>, <var data-var='time'>19:11</var> UTC Monitoring - GitHub has posted an update indicating recovery of the aforementioned issue. We continue monitoring Semaphore closely as things stabilize. (GitHub Status: https://www.githubstatus.com/incidents/f314nlctbfs5) Feb <var data-var='date'> 3</var>, <var data-var='time'>17:34</var> UTC Identified - Webhook delivery delays between GitHub and Semaphore appear to be caused by degraded performance on GitHub’s side. GitHub is currently reporting an active incident that appears related. We are monitoring the situation closely and will provide updates as GitHub resolves the issue. Feb <var data-var='date'> 3</var>, <var data-var='time'>15:46</var> UTC Investigating - We are currently investigating this issue.

Delivering webhooks delays between Github and Semaphore

Feb <var data-var='date'> 3</var>, <var data-var='time'>19:24</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 3</var>, <var data-var='time'>18:52</var> UTC Monitoring - We’re seeing considerable improvement in internal performance metrics, and we will continue monitoring until conditions fully stabilize. Feb <var data-var='date'> 3</var>, <var data-var='time'>18:06</var> UTC Investigating - Webhook delivery delays from GitHub led to a backlog of workflows being triggered all at once as delivery resumes. This caused a temporary spike in demand for some Semaphore machines. We are actively working to reduce the impact and restore normal capacity. We continue monitoring both GitHub’s incident and Semaphore infrastructure closely, and we will provide further updates as conditions stabilize.

Delays in Job Processing

Feb <var data-var='date'> 1</var>, <var data-var='time'>11:00</var> UTC Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 1</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>30</var>, <var data-var='time'>08:32</var> UTC Scheduled - We will be undergoing scheduled maintenance during this time.

Scheduled database maintenance

Jan <var data-var='date'>27</var>, <var data-var='time'>19:10</var> UTC Resolved - This incident has been resolved. Jan <var data-var='date'>27</var>, <var data-var='time'>18:49</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jan <var data-var='date'>27</var>, <var data-var='time'>17:36</var> UTC Update - We are continuing to work on a fix for this issue. Jan <var data-var='date'>27</var>, <var data-var='time'>16:35</var> UTC Update - We are continuing to work on a fix for this issue. Jan <var data-var='date'>27</var>, <var data-var='time'>15:34</var> UTC Identified - The issue has been identified and a fix is being implemented.

Delays in job processing on Semaphore

Jan <var data-var='date'>18</var>, <var data-var='time'>11:00</var> UTC Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>18</var>, <var data-var='time'>10:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>14</var>, <var data-var='time'>13:36</var> UTC Scheduled - We will be undergoing scheduled maintenance during this time.

Scheduled database server maintenance

Nov <var data-var='date'>12</var>, <var data-var='time'>18:10</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'>12</var>, <var data-var='time'>17:34</var> UTC Monitoring - The connection seems stable and we keep monitoring it. Nov <var data-var='date'>12</var>, <var data-var='time'>16:24</var> UTC Identified - We’ve identified that these issues are being caused by inconsistent network routing between Europe and the United States on upstream providers. Semaphore’s systems remain healthy, but affected routes can cause slower file transfer and occasional connection timeouts. Nov <var data-var='date'>12</var>, <var data-var='time'>15:28</var> UTC Update - We are continuing to investigate this issue. Nov <var data-var='date'>12</var>, <var data-var='time'>14:55</var> UTC Investigating - We are currently investigating the issue. Implementing the retry command should help mitigate the issue: https://docs.semaphore.io/reference/toolbox#retry

Network connectivity issues towards us-east

Nov <var data-var='date'> 7</var>, <var data-var='time'>07:15</var> UTC Resolved - We have observed connection times recover and stay healthy for the affected regions. Nov <var data-var='date'> 7</var>, <var data-var='time'>02:29</var> UTC Update - We are continuing to monitor the AWS US-East network congestion. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 7</var>, <var data-var='time'>01:48</var> UTC Update - We are continuing to monitor the AWS US-East region slowness. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 7</var>, <var data-var='time'>01:07</var> UTC Update - Increased network congestion detected in the AWS US-East region. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 7</var>, <var data-var='time'>00:36</var> UTC Update - Our team is still monitoring the AWS US-East region performance. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 7</var>, <var data-var='time'>00:03</var> UTC Update - Our team is following up the slight recovery in AWS US-East region. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>23:30</var> UTC Update - We are continuing to follow up the AWS US-East region slowness issue. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>23:02</var> UTC Update - We are continuing to monitor the AWS US-East region slowness. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>22:31</var> UTC Update - Performance in AWS US-East region is showing gradual recovery. Our team continues to monitor the situation. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>21:40</var> UTC Update - Our team is still monitoring the AWS US-East region slowness. Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>21:02</var> UTC Update - Connections to us-west, ap-northeast, and GCP are performing normally. If your workflows are affected, pushing/pulling data to and from these regions should help mitigate the issue. We are continuing to monitor the AWS US-East region slowness. Nov <var data-var='date'> 6</var>, <var data-var='time'>20:49</var> UTC Update - We have observed slight improvements in the operations towards the AWS US-East region. We are continuing to work on this issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>19:58</var> UTC Update - We are continuing to investigate this issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>19:05</var> UTC Update - We are continuing to investigate this issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>18:33</var> UTC Update - Some operations in the AWS US-East region may be experiencing slow performance. Our team is still investigating. Nov <var data-var='date'> 6</var>, <var data-var='time'>17:50</var> UTC Update - We are continuing to investigate this issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>17:22</var> UTC Update - We are continuing to investigate this issue. Nov <var data-var='date'> 6</var>, <var data-var='time'>16:51</var> UTC Investigating - Operations towards the AWS US-East region might present some slowness. We are currently investigating this issue.

Network congestion on route to AWS ECR us-east region

Nov <var data-var='date'> 5</var>, <var data-var='time'>00:00</var> UTC Resolved - This incident has been resolved. Nov <var data-var='date'> 4</var>, <var data-var='time'>23:16</var> UTC Update - Our team is still monitoring the AWS US-East region levels to ensure the issue is resolved. Nov <var data-var='date'> 4</var>, <var data-var='time'>22:25</var> UTC Update - We are continuing to monitor for any further issues. Nov <var data-var='date'> 4</var>, <var data-var='time'>21:05</var> UTC Update - We are continuing to monitor for any further issues. Nov <var data-var='date'> 4</var>, <var data-var='time'>20:01</var> UTC Monitoring - Operations towards the AWS US-East region should be back to normal. We are still monitoring for any further issues. Nov <var data-var='date'> 4</var>, <var data-var='time'>19:09</var> UTC Update - We are continuing to investigate this issue. Nov <var data-var='date'> 4</var>, <var data-var='time'>17:52</var> UTC Update - We are continuing to investigate this issue. Nov <var data-var='date'> 4</var>, <var data-var='time'>17:19</var> UTC Investigating - We are currently investigating this issue. Operations towards the AWS US-East region might present some slowness.

Oct <var data-var='date'>29</var>, <var data-var='time'>16:28</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>29</var>, <var data-var='time'>15:58</var> UTC Update - We are continuing to monitor for any further issues. Oct <var data-var='date'>29</var>, <var data-var='time'>15:22</var> UTC Update - We are continuing to monitor for any further issues. Oct <var data-var='date'>29</var>, <var data-var='time'>14:22</var> UTC Update - We are continuing to monitor for any further issues. Oct <var data-var='date'>29</var>, <var data-var='time'>13:30</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Oct <var data-var='date'>29</var>, <var data-var='time'>12:53</var> UTC Update - We are continuing to work on a fix for this issue. Oct <var data-var='date'>29</var>, <var data-var='time'>12:24</var> UTC Update - We are continuing to work on a fix for this issue. Oct <var data-var='date'>29</var>, <var data-var='time'>11:38</var> UTC Identified - The issue has been identified and a fix is being implemented.

Slow job processing

Oct <var data-var='date'>21</var>, <var data-var='time'>09:58</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>21</var>, <var data-var='time'>09:44</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Oct <var data-var='date'>21</var>, <var data-var='time'>09:35</var> UTC Identified - The issue has been identified, and we are working on a fix. Oct <var data-var='date'>21</var>, <var data-var='time'>08:58</var> UTC Investigating - We are currently investigating this issue.

Delays in job completion on Semaphore

Oct <var data-var='date'>21</var>, <var data-var='time'>01:00</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>21</var>, <var data-var='time'>00:44</var> UTC Monitoring - Issues with image push/pulls seem to be resolved. We are still closely monitoring the situation. Oct <var data-var='date'>20</var>, <var data-var='time'>23:52</var> UTC Update - Image push/pulls may be affected by the ongoing Docker Hub outage. We continue to monitor the situation. Oct <var data-var='date'>20</var>, <var data-var='time'>21:38</var> UTC Update - Some image push/pulls seem to be affected by the ongoing Docker Hub outage. Our team is monitoring the situation. Oct <var data-var='date'>20</var>, <var data-var='time'>19:44</var> UTC Update - Image push/pulls seem to be affected by the ongoing Docker Hub outage. We keep monitoring the situation. Oct <var data-var='date'>20</var>, <var data-var='time'>18:01</var> UTC Update - Image push/pulls seem to be affected again by the ongoing Docker Hub outage. We continue to monitor the situation. Oct <var data-var='date'>20</var>, <var data-var='time'>16:01</var> UTC Identified - Image push/pulls seem to be affected again by the ongoing Docker Hub outage. We continue to monitor the situation. Oct <var data-var='date'>20</var>, <var data-var='time'>14:40</var> UTC Monitoring - Issues with image push/pulls seem to be resolved. We are still closely monitoring the situation. Oct <var data-var='date'>20</var>, <var data-var='time'>14:04</var> UTC Update - We are continuing to monitor the situation: - Image push/pulls are still being affected by the Docker Hub outage. - Broader AWS service issues could still indirectly affect some users' workloads. Oct <var data-var='date'>20</var>, <var data-var='time'>12:10</var> UTC Update - We are continuing to monitor the situation: - Image push/pulls are still being affected by the Docker Hub outage. - Broader AWS service issues could still indirectly affect some users' workloads. Oct <var data-var='date'>20</var>, <var data-var='time'>10:29</var> UTC Update - Image push/pulls are still being affected by the Docker Hub outage. We are closely monitoring the situation. We are continuing to monitor broader AWS service issues, which could still indirectly affect some users' workloads. Oct <var data-var='date'>20</var>, <var data-var='time'>08:44</var> UTC Identified - Docker Hub is experiencing an outage, which is affecting image push/pulls in CI. We are also monitoring broader AWS service issues, which could indirectly affect some users' workloads.

Docker Hub outage affecting image push/pulls in Semaphore

Oct <var data-var='date'>15</var>, <var data-var='time'>00:31</var> UTC Resolved - This incident has been resolved. Oct <var data-var='date'>15</var>, <var data-var='time'>00:00</var> UTC Monitoring - We are in active communication with Semaphore’s upstream provider to bring this issue to full resolution. Oct <var data-var='date'>14</var>, <var data-var='time'>22:02</var> UTC Update - We are in active communication with Semaphore's server provider, and we are continuing to investigate the issue. Oct <var data-var='date'>14</var>, <var data-var='time'>21:29</var> UTC Update - We are in active communication with Semaphore's server provider, and we are continuing to investigate the issue. Oct <var data-var='date'>14</var>, <var data-var='time'>20:56</var> UTC Update - We are continuing to investigate this issue. Oct <var data-var='date'>14</var>, <var data-var='time'>20:26</var> UTC Update - We are continuing to investigate this issue. Oct <var data-var='date'>14</var>, <var data-var='time'>19:47</var> UTC Investigating - We are currently investigating this issue.

Is Semaphore CI down?

Semaphore CI status is UP

Lastest published events

Delays when checking out GitHub repos via SSH

Delays when checking out GitHub repos via SSH

Delays in job processing

Semaphore CI Services, Components and Regions

Recent Semaphore CI events

Delays when checking out GitHub repos via SSH

Delays when checking out GitHub repos via SSH

Delays in job processing

Delays in job processing

Delays in job processing

Delays in job processing

Get your FREE account now,
5 monitors included!

Is Semaphore CI down?

Semaphore CI status is UP

Lastest published events

Delays when checking out GitHub repos via SSH

Delays when checking out GitHub repos via SSH

Delays in job processing

Semaphore CI Services, Components and Regions

Recent Semaphore CI events

Delays when checking out GitHub repos via SSH

Delays when checking out GitHub repos via SSH

Delays in job processing

Delays in job processing

Delays in job processing

Delays in job processing

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!