Status: Resolved All systems are operational.

Elevated queue time for a small set of GitHub Action jobs

Elevated error rate from GitHub API

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Social SSO (Operational)</li>
<li>SAML SSO (Operational)</li>
 </ul>

Some IAM calls are failing

Status: Resolved We've processed the backlog of jobs. 

All systems are operational. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
 </ul>

Increased queue time for subset of GitHub jobs

Status: Resolved We've confirmed the fix resolves the root cause and we're seeing full recovery. Affected components
 <ul>
 <li>Devboxes (Operational)</li>
<li>Social SSO (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Artifacts (Operational)</li>
<li>SAML SSO (Operational)</li>
 </ul>

Elevated error rate in our control plane

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Dashboard (Operational)</li>
<li>Logging (Operational)</li>
 </ul>

Elevated errors during logs serving

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>MacOS arm64 (Operational)</li>
 </ul>

Delayed startup for macOS instance

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Container Registry (Operational)</li>
<li>Previews (Operational)</li>
<li>Turborepo (Operational)</li>
<li>Gradle (Operational)</li>
<li>Dashboard (Operational)</li>
<li>Illinois, US (Operational)</li>
<li>Artifacts (Operational)</li>
<li>Windows amd64 (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Linux arm64 on Apple Silicon (Operational)</li>
<li>Metrics (Operational)</li>
<li>Devboxes (Operational)</li>
<li>Linux arm64 (Operational)</li>
<li>Bazel (Operational)</li>
<li>MacOS arm64 (Operational)</li>
<li>Support (Operational)</li>
<li>HTTP Cache (Operational)</li>
<li>Social SSO (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Linux amd64 (Operational)</li>
<li>Northern Virginia, US (Operational)</li>
<li>Partner UI (Operational)</li>
<li>Logging (Operational)</li>
<li>Western Europe (Operational)</li>
 </ul>

Increased error rate in a single compute shard

Increased error rate from GitHub

Elevated error rate from Dockerhub

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Container Registry (Operational)</li>
 </ul>

Registry upload failures in Virginia

Elevated GitHub action error rate

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Logging (Operational)</li>
<li>Metrics (Operational)</li>
 </ul>

Partial unavailability of instance metrics

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
 </ul>

Elevated GitHub actions queue time

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Artifacts (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Devboxes (Operational)</li>
<li>Social SSO (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Container Registry (Operational)</li>
 </ul>

Elevated error rate

Elevated queue time for GitHub action jobs

Elevated queue time for GitHub Actions

Elevated queue time for a subset of GitHub jobs

Elevated queue time for a subset of GitHub actions jobs

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Social SSO (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Linux arm64 (Operational)</li>
<li>Linux amd64 (Operational)</li>
<li>MacOS arm64 (Operational)</li>
<li>Windows amd64 (Operational)</li>
<li>Metrics (Operational)</li>
<li>Gradle (Operational)</li>
<li>Dashboard (Operational)</li>
<li>Northern Virginia, US (Operational)</li>
<li>Artifacts (Operational)</li>
<li>Support (Operational)</li>
<li>Partner UI (Operational)</li>
<li>Linux arm64 on Apple Silicon (Operational)</li>
<li>Western Europe (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Illinois, US (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Previews (Operational)</li>
<li>Devboxes (Operational)</li>
<li>Bazel (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Turborepo (Operational)</li>
<li>Logging (Operational)</li>
<li>HTTP Cache (Operational)</li>
 </ul>

Delayed webhook deliveries from GitHub leading to queue time

GitHub outage leading to job queuing

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>SAML SSO (Operational)</li>
 </ul>

SSO login experiencing errors

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Artifacts (Operational)</li>
<li>Previews (Operational)</li>
<li>Devboxes (Operational)</li>
 </ul>

Impact to multiple services

Status: Resolved All connections to the unhealthy instance were terminated, and error rates fully recovered. Affected components
 <ul>
 <li>Northern Virginia, US (Operational)</li>
<li>Previews (Operational)</li>
<li>Social SSO (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Illinois, US (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Linux amd64 (Operational)</li>
<li>Artifacts (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Support (Operational)</li>
<li>Linux arm64 on Apple Silicon (Operational)</li>
<li>HTTP Cache (Operational)</li>
<li>Devboxes (Operational)</li>
<li>MacOS arm64 (Operational)</li>
<li>Western Europe (Operational)</li>
<li>Linux arm64 (Operational)</li>
<li>Partner UI (Operational)</li>
<li>Turborepo (Operational)</li>
<li>Gradle (Operational)</li>
<li>Dashboard (Operational)</li>
<li>Bazel (Operational)</li>
<li>Windows amd64 (Operational)</li>
<li>Logging (Operational)</li>
<li>Metrics (Operational)</li>
 </ul>

Increased error rate for multiple services

Status: Resolved We're seeing recovery across the board, with runners no longer failing to acquire jobs. Queue time has returned to normal across all jobs.

Increased error rate from GitHub resulting in increased queue time

Status: Resolved We've applied a mitigation and are seeing full recovery. Affected components
 <ul>
 <li>Container Registry (Operational)</li>
<li>Devboxes (Operational)</li>
 </ul>

Failure to read blobs in one of the Virginia locations

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Northern Virginia, US (Operational)</li>
 </ul>

Increase in GitHub queue time

Increase in queue time across GitHub jobs

Elevated error rate for container registry

Status: Resolved We've confirmed recovery, jobs are being picked up as normal. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
 </ul>

Increased queue time for GitHub Jobs

Invalid GitHub events leading to job queuing

Elevated error rate in GitHub event ingestion

Delayed webhook deliveries from GitHub

Status: Resolved We're seeing delays from GitHub going down, and almost all jobs are now starting without increased latency. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
 </ul>

Delayed webhook delivery from GitHub resulting in increased queue time

Status: Resolved All systems are operational. A change performed during a maintenance window was rolled back. Affected components
 <ul>
 <li>Linux arm64 (Operational)</li>
<li>Northern Virginia, US (Operational)</li>
<li>Artifacts (Operational)</li>
<li>Partner UI (Operational)</li>
<li>Logging (Operational)</li>
<li>HTTP Cache (Operational)</li>
<li>Western Europe (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Illinois, US (Operational)</li>
<li>MacOS arm64 (Operational)</li>
<li>Turborepo (Operational)</li>
<li>Metrics (Operational)</li>
<li>Gradle (Operational)</li>
<li>Dashboard (Operational)</li>
<li>Linux amd64 (Operational)</li>
<li>Bazel (Operational)</li>
<li>Previews (Operational)</li>
<li>Windows amd64 (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Container Registry (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Support (Operational)</li>
<li>Linux arm64 on Apple Silicon (Operational)</li>
<li>Devboxes (Operational)</li>
<li>Social SSO (Operational)</li>
 </ul>

Queue time during maintenance window

Status: Resolved We're seeing delays from GitHub going down, and almost all jobs are now starting without increased latency.

Status: Resolved New jobs are starting as normal.

Jobs started during the period of degraded deliveries are slowly getting picked up as GitHub is working its way through the backlog.

Delayed job notifications from GitHub causing increased queue time

Status: Resolved We applied a mitigation to prioritize a different issuer, certificate issuing has recovered. Affected components
 <ul>
 <li>Previews (Operational)</li>
 </ul>

Subset of certificates for Previews are failing to be issued

Status: Resolved We've confirmed the alternative transit path has fully resolved the connectivity issues. Affected components
 <ul>
 <li>Western Europe (Operational)</li>
 </ul>

Degraded connectivity to GitHub in Europe

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Illinois, US (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Logging (Operational)</li>
<li>Metrics (Operational)</li>
<li>Previews (Operational)</li>
<li>Dashboard (Operational)</li>
<li>Western Europe (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Linux arm64 (Operational)</li>
<li>Linux amd64 (Operational)</li>
<li>Northern Virginia, US (Operational)</li>
<li>Artifacts (Operational)</li>
<li>Social SSO (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>MacOS arm64 (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Support (Operational)</li>
<li>Turborepo (Operational)</li>
<li>Gradle (Operational)</li>
<li>Devboxes (Operational)</li>
<li>Bazel (Operational)</li>
<li>Windows amd64 (Operational)</li>
<li>Partner UI (Operational)</li>
<li>Linux arm64 on Apple Silicon (Operational)</li>
<li>HTTP Cache (Operational)</li>
 </ul>

Increased error rate and queue time

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Social SSO (Operational)</li>
 </ul>

Degraded performance of social SSO logins

Status: Resolved We confirm seeing full recovery https://www.githubstatus.com/incidents/gnftqj9htp0g

Is Namespace down?

Namespace status is UP

Lastest published events

Elevated queue time for a small set of GitHub Action jobs

Elevated error rate from GitHub API

Some IAM calls are failing

Namespace Services, Components and Regions

Recent Namespace events

Elevated queue time for a small set of GitHub Action jobs

Elevated error rate from GitHub API

Some IAM calls are failing

Increased queue time for subset of GitHub jobs

Elevated error rate in our control plane

Elevated errors during logs serving