THIS IS A SCHEDULED EVENT Aug <var data-var='date'>22</var>, <var data-var='time'>09:00</var> - <var data-var='time'>11:00</var> UTC Aug <var data-var='date'> 7</var>, <var data-var='time'>13:07</var> UTC Scheduled - The platform may experience degraded performance during the maintenance window, including up to 15 minutes of downtime.

Scheduled Semaphore Maintenance

Aug <var data-var='date'>21</var>, <var data-var='time'>20:47</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>21</var>, <var data-var='time'>20:27</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>21</var>, <var data-var='time'>20:03</var> UTC Update - We are continuing to work on a fix for this issue. Aug <var data-var='date'>21</var>, <var data-var='time'>19:26</var> UTC Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>21</var>, <var data-var='time'>19:14</var> UTC Investigating - We are currently investigating this issue.

Issues with Bitbucket integration

Aug <var data-var='date'>18</var>, <var data-var='time'>17:47</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'>18</var>, <var data-var='time'>17:16</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>18</var>, <var data-var='time'>16:28</var> UTC Update - We are continuing to work on a fix for this issue. Aug <var data-var='date'>18</var>, <var data-var='time'>15:28</var> UTC Identified - The issue is still ongoing, and the engineering team is actively working on a fix. Aug <var data-var='date'>18</var>, <var data-var='time'>15:24</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'>18</var>, <var data-var='time'>15:18</var> UTC Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>18</var>, <var data-var='time'>15:15</var> UTC Update - Jobs configured to run on F1 machines may take longer than usual to start. Aug <var data-var='date'>18</var>, <var data-var='time'>15:07</var> UTC Investigating - We are currently investigating this issue.

Delays in job processing - F1 machines

Aug <var data-var='date'> 4</var>, <var data-var='time'>12:04</var> UTC Resolved - This incident has been resolved. Aug <var data-var='date'> 4</var>, <var data-var='time'>11:30</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'> 4</var>, <var data-var='time'>10:38</var> UTC Identified - The issue has been identified and a fix is being implemented.

Bitbucket integration may currently be experiencing issues

Jul <var data-var='date'>23</var>, <var data-var='time'>02:18</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>23</var>, <var data-var='time'>02:00</var> UTC Monitoring - A fix has been implemented and we have observed an improvement in job processing. We are continuing to monitor the situation. Jul <var data-var='date'>23</var>, <var data-var='time'>01:25</var> UTC Identified - The issue has been identified and a fix is being implemented. Jul <var data-var='date'>23</var>, <var data-var='time'>01:03</var> UTC Update - We are continuing to investigate this issue. Jul <var data-var='date'>23</var>, <var data-var='time'>00:35</var> UTC Investigating - We are currently investigating this issue.

Delays in job processing

Jul <var data-var='date'>21</var>, <var data-var='time'>13:46</var> UTC Resolved - GitHub confirmed that the incident on their end has been resolved. You can review the GitHub update here: https://www.githubstatus.com/incidents/g40zcbvchny4 Jul <var data-var='date'>21</var>, <var data-var='time'>12:18</var> UTC Update - GitHub confirmed that the incident on their end has been resolved, and services should now be returning to normal. We are continuing to monitor the situation. You can review the GitHub update here: https://www.githubstatus.com/incidents/g40zcbvchny4 Jul <var data-var='date'>21</var>, <var data-var='time'>10:53</var> UTC Update - We are continuing to monitor the situation, and you can follow the current GitHub incident at the following link: https://www.githubstatus.com/incidents/g40zcbvchny4 Jul <var data-var='date'>21</var>, <var data-var='time'>09:25</var> UTC Monitoring - Our team is monitoring the situation. If you are affected, please regenerate your deploy key by following the instructions on this page: https://docs.semaphore.io/using-semaphore/connect-github#deploy-key Jul <var data-var='date'>21</var>, <var data-var='time'>09:06</var> UTC Update - Our engineering team is still investigating and working on this issue, but as a current workaround, you can regenerate your GitHub deployment key. You can find the instructions on how to do that here: https://docs.semaphore.io/using-semaphore/connect-github#deploy-key Jul <var data-var='date'>21</var>, <var data-var='time'>08:49</var> UTC Update - As a workaround to get more consistent builds during this time, please try adding the --use-cache flag to your checkout command. Read more about it here: https://docs.semaphore.io/reference/toolbox#cache-full-clone Jul <var data-var='date'>21</var>, <var data-var='time'>08:36</var> UTC Investigating - We are currently investigating this issue.

Some jobs may have issues authenticating with GitHub

Jul <var data-var='date'>17</var>, <var data-var='time'>13:25</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>17</var>, <var data-var='time'>10:51</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>17</var>, <var data-var='time'>10:32</var> UTC Investigating - We are currently investigating this issue.

Some features may currently be unresponsive

Jul <var data-var='date'>17</var>, <var data-var='time'>01:05</var> UTC Resolved - This incident has been resolved. Jul <var data-var='date'>17</var>, <var data-var='time'>00:43</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'>17</var>, <var data-var='time'>00:21</var> UTC Investigating - We are currently investigating this issue.

Jun <var data-var='date'>24</var>, <var data-var='time'>21:26</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>24</var>, <var data-var='time'>21:16</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>24</var>, <var data-var='time'>21:07</var> UTC Identified - The issue has been identified and a fix is being implemented. Jun <var data-var='date'>24</var>, <var data-var='time'>21:01</var> UTC Investigating - We are currently investigating this issue.

Elevated error rates

Jun <var data-var='date'>10</var>, <var data-var='time'>16:22</var> UTC Resolved - This incident has been resolved. Jun <var data-var='date'>10</var>, <var data-var='time'>16:14</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun <var data-var='date'>10</var>, <var data-var='time'>15:20</var> UTC Update - The issue has been identified, and our team is implementing a fix. Jun <var data-var='date'>10</var>, <var data-var='time'>15:12</var> UTC Identified - We're experiencing an elevated level of 500's errors and are currently looking into the issue.

Elevated Error Rates

May <var data-var='date'>20</var>, <var data-var='time'>15:32</var> UTC Resolved - This incident has been resolved. May <var data-var='date'>20</var>, <var data-var='time'>15:02</var> UTC Monitoring - A fix has been implemented, and everything should be back to normal. The previously shared snippets can be removed. May <var data-var='date'>20</var>, <var data-var='time'>13:59</var> UTC Identified - We identified slow GitHub operations related to problems with the network, and we deployed a fix to re-route those operations to a healthier server. Some HTTPS operations started to fail because of that. To disable the fix, you should use the command: sudo sed -i 's/^\([0-9].*github\.com\)/#\1/' /etc/hosts And to re-enable, you can use: sudo sed -i 's/^#\([0-9].*github\.com\)/\1/' /etc/hosts May <var data-var='date'>20</var>, <var data-var='time'>12:48</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. May <var data-var='date'>20</var>, <var data-var='time'>08:44</var> UTC Identified - We identified slow communication between Semaphore jobs and GitHub. We are continuing to work with our upstream provider to resolve this. For impacted jobs, please replace the normal checkout command with the following: retry --times 5 "timeout 40 bash -lc 'source ~/.toolbox/toolbox; checkout'" And afterwards, use cd to change the directory to the cloned repository (replace with your project).

Delays when checking out GitHub repos via SSH

May <var data-var='date'>19</var>, <var data-var='time'>15:42</var> UTC Resolved - This incident has been resolved. May <var data-var='date'>19</var>, <var data-var='time'>15:10</var> UTC Monitoring - We are not currently seeing timeouts related to slow connections. Our team is continuing to monitor the situation to confirm that service has fully returned to normal. May <var data-var='date'>19</var>, <var data-var='time'>11:18</var> UTC Identified - We identified that the issue is slow communication between our jobs and GitHub. We are currently working with our upstream provider to to resolve this. For impacted jobs, please replace the normal checkout command with the following: retry --times 5 "timeout 40 bash -lc 'source ~/.toolbox/toolbox; checkout'" And afterwards, use cd <dir> to change the directory to the cloned repository (replace <dir> with your project). May <var data-var='date'>19</var>, <var data-var='time'>10:38</var> UTC Update - We are continuing to investigate this issue. May <var data-var='date'>19</var>, <var data-var='time'>10:00</var> UTC Investigating - We are currently investigating this issue.

Mar <var data-var='date'>12</var>, <var data-var='time'>17:45</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>12</var>, <var data-var='time'>17:11</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'>12</var>, <var data-var='time'>17:03</var> UTC Identified - The issue has been identified and a fix is being implemented. Mar <var data-var='date'>12</var>, <var data-var='time'>15:55</var> UTC Investigating - We are currently investigating this issue.

Mar <var data-var='date'>11</var>, <var data-var='time'>17:42</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'>11</var>, <var data-var='time'>17:18</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'>11</var>, <var data-var='time'>16:28</var> UTC Identified - The issue has been identified and a fix is being implemented.

Mar <var data-var='date'> 8</var>, <var data-var='time'>18:52</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 8</var>, <var data-var='time'>18:22</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'> 8</var>, <var data-var='time'>17:26</var> UTC Identified - The issue has been identified and a fix is being implemented. Mar <var data-var='date'> 8</var>, <var data-var='time'>16:25</var> UTC Update - We are continuing to investigate this issue. Mar <var data-var='date'> 8</var>, <var data-var='time'>15:21</var> UTC Investigating - We are currently investigating this issue.

Mar <var data-var='date'> 3</var>, <var data-var='time'>18:05</var> UTC Resolved - This incident has been resolved. Mar <var data-var='date'> 3</var>, <var data-var='time'>17:36</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar <var data-var='date'> 3</var>, <var data-var='time'>17:06</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'>24</var>, <var data-var='time'>10:32</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>24</var>, <var data-var='time'>10:05</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. We will provide our next update within 30 minutes or sooner if things change. Feb <var data-var='date'>24</var>, <var data-var='time'>09:54</var> UTC Identified - The issue has been identified and we are working on a fix. We will provide our next update within 30 minutes or sooner if things change. We thank you for your patience while we continue to work to resolve this issue. Feb <var data-var='date'>24</var>, <var data-var='time'>09:34</var> UTC Update - We are continuing to investigate job start failures and processing delays. We will continue to update while investigating this. Feb <var data-var='date'>24</var>, <var data-var='time'>09:06</var> UTC Update - We are continuing to investigate this issue. Feb <var data-var='date'>24</var>, <var data-var='time'>08:49</var> UTC Investigating - We are currently investigating this issue.

Jobs Not Starting and Processing Delays

Feb <var data-var='date'>20</var>, <var data-var='time'>18:43</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>20</var>, <var data-var='time'>18:17</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>20</var>, <var data-var='time'>17:30</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>20</var>, <var data-var='time'>15:56</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'>18</var>, <var data-var='time'>20:41</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>18</var>, <var data-var='time'>20:12</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>18</var>, <var data-var='time'>19:12</var> UTC Update - Our team is still working on a fix for this issue. Feb <var data-var='date'>18</var>, <var data-var='time'>18:18</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>18</var>, <var data-var='time'>17:07</var> UTC Identified - The issue has been identified and a fix is being implemented. Feb <var data-var='date'>18</var>, <var data-var='time'>16:13</var> UTC Investigating - We are currently investigating this issue.

Feb <var data-var='date'>17</var>, <var data-var='time'>18:18</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'>17</var>, <var data-var='time'>17:50</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>17</var>, <var data-var='time'>17:45</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>17</var>, <var data-var='time'>16:34</var> UTC Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>17</var>, <var data-var='time'>16:32</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'> 7</var>, <var data-var='time'>11:00</var> UTC Completed - The scheduled maintenance has been completed. Feb <var data-var='date'> 7</var>, <var data-var='time'>09:00</var> UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'> 3</var>, <var data-var='time'>13:56</var> UTC Scheduled - Semaphore will be temporarily unavailable during this scheduled maintenance on Saturday.

Infrastructure maintenance

Feb <var data-var='date'> 5</var>, <var data-var='time'>17:28</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 5</var>, <var data-var='time'>16:28</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'> 4</var>, <var data-var='time'>18:10</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 4</var>, <var data-var='time'>17:40</var> UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'> 4</var>, <var data-var='time'>16:38</var> UTC Identified - The issue has been identified and a fix is being implemented.

Feb <var data-var='date'> 3</var>, <var data-var='time'>19:50</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 3</var>, <var data-var='time'>19:11</var> UTC Monitoring - GitHub has posted an update indicating recovery of the aforementioned issue. We continue monitoring Semaphore closely as things stabilize. (GitHub Status: https://www.githubstatus.com/incidents/f314nlctbfs5) Feb <var data-var='date'> 3</var>, <var data-var='time'>17:34</var> UTC Identified - Webhook delivery delays between GitHub and Semaphore appear to be caused by degraded performance on GitHub’s side. GitHub is currently reporting an active incident that appears related. We are monitoring the situation closely and will provide updates as GitHub resolves the issue. Feb <var data-var='date'> 3</var>, <var data-var='time'>15:46</var> UTC Investigating - We are currently investigating this issue.

Delivering webhooks delays between Github and Semaphore

Feb <var data-var='date'> 3</var>, <var data-var='time'>19:24</var> UTC Resolved - This incident has been resolved. Feb <var data-var='date'> 3</var>, <var data-var='time'>18:52</var> UTC Monitoring - We’re seeing considerable improvement in internal performance metrics, and we will continue monitoring until conditions fully stabilize. Feb <var data-var='date'> 3</var>, <var data-var='time'>18:06</var> UTC Investigating - Webhook delivery delays from GitHub led to a backlog of workflows being triggered all at once as delivery resumes. This caused a temporary spike in demand for some Semaphore machines. We are actively working to reduce the impact and restore normal capacity. We continue monitoring both GitHub’s incident and Semaphore infrastructure closely, and we will provide further updates as conditions stabilize.

Is Semaphore CI down?

Semaphore CI status is UP

Lastest published events

Issues with Bitbucket integration

Delays in job processing - F1 machines

Bitbucket integration may currently be experiencing issues

Semaphore CI Services, Components and Regions

Recent Semaphore CI events

Scheduled Semaphore Maintenance

Issues with Bitbucket integration

Delays in job processing - F1 machines

Bitbucket integration may currently be experiencing issues

Delays in job processing

Some jobs may have issues authenticating with GitHub