Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Devboxes (Operational)</li>
 </ul>

A subset of devboxes is failing to start

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Artifacts (Operational)</li>
<li>Container Registry (Operational)</li>
 </ul>

Degraded registry and artifact storage performance

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>SAML SSO (Operational)</li>
 </ul>

SAML SSO logins are failing

Status: Resolved All jobs are being processed normally. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
 </ul>

Delayed runner startup

Status: Resolved Error rates have recovered, all systems operational. Affected components
 <ul>
 <li>Turborepo (Operational)</li>
<li>Gradle (Operational)</li>
<li>HTTP Cache (Operational)</li>
<li>Illinois, US (Operational)</li>
<li>Bazel (Operational)</li>
<li>Artifacts (Operational)</li>
 </ul>

Small subset of artifact and cache requests in Illinois region failing

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Western Europe (Operational)</li>
<li>Devboxes (Operational)</li>
 </ul>

Elevated instance creation time

Status: Resolved We've confirmed applied mitigations have lead to full recovery of the affected compute shard. Affected components
 <ul>
 <li>Western Europe (Operational)</li>
<li>Devboxes (Operational)</li>
 </ul>

Elevated instance creation latency

Status: Resolved The fix addressed the root cause and all images are being served correctly.

Subset of container image pulls from external registries failing

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Dashboard (Operational)</li>
<li>Partner UI (Operational)</li>
 </ul>

Ongoing maintenance

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Northern Virginia, US (Operational)</li>
 </ul>

Increase in queue time of jobs in Virginia (iad-2 and iad-3)

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
 </ul>

Extended GitHub queue time

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Dashboard (Operational)</li>
 </ul>

Instance pages failing to load

We're investigating reports of GitHub connectivity issues

Network disruption in iad2

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Artifacts (Operational)</li>
<li>Previews (Operational)</li>
<li>Devboxes (Operational)</li>
 </ul>

A subset of control plane calls were failing, including job starts

Elevated queue time for GitHub action jobs

Network availability

Status: Resolved All systems are operational.

Elevated queue time for a small set of GitHub Action jobs

Elevated error rate from GitHub API

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Social SSO (Operational)</li>
<li>SAML SSO (Operational)</li>
 </ul>

Some IAM calls are failing

Status: Resolved We've processed the backlog of jobs. 

All systems are operational. Affected components
 <ul>
 <li>GitHub Runners (Operational)</li>
 </ul>

Increased queue time for subset of GitHub jobs

Status: Resolved We've confirmed the fix resolves the root cause and we're seeing full recovery. Affected components
 <ul>
 <li>Social SSO (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Artifacts (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Devboxes (Operational)</li>
 </ul>

Elevated error rate in our control plane

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Logging (Operational)</li>
<li>Dashboard (Operational)</li>
 </ul>

Elevated errors during logs serving

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>MacOS arm64 (Operational)</li>
 </ul>

Delayed startup for macOS instance

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Turborepo (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Artifacts (Operational)</li>
<li>HTTP Cache (Operational)</li>
<li>Illinois, US (Operational)</li>
<li>Linux amd64 (Operational)</li>
<li>Linux arm64 on Apple Silicon (Operational)</li>
<li>Logging (Operational)</li>
<li>Metrics (Operational)</li>
<li>Devboxes (Operational)</li>
<li>Social SSO (Operational)</li>
<li>Western Europe (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>MacOS arm64 (Operational)</li>
<li>Previews (Operational)</li>
<li>Support (Operational)</li>
<li>Gradle (Operational)</li>
<li>Dashboard (Operational)</li>
<li>Linux arm64 (Operational)</li>
<li>Bazel (Operational)</li>
<li>Northern Virginia, US (Operational)</li>
<li>Windows amd64 (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Partner UI (Operational)</li>
 </ul>

Increased error rate in a single compute shard

Increased error rate from GitHub

Elevated error rate from Dockerhub

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Container Registry (Operational)</li>
 </ul>

Registry upload failures in Virginia

Elevated GitHub action error rate

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Logging (Operational)</li>
<li>Metrics (Operational)</li>
 </ul>

Partial unavailability of instance metrics

Elevated GitHub actions queue time

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Social SSO (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Artifacts (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Devboxes (Operational)</li>
 </ul>

Elevated error rate

Elevated queue time for GitHub Actions

Elevated queue time for a subset of GitHub jobs

Elevated queue time for a subset of GitHub actions jobs

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Social SSO (Operational)</li>
<li>Illinois, US (Operational)</li>
<li>Linux amd64 (Operational)</li>
<li>MacOS arm64 (Operational)</li>
<li>Previews (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Turborepo (Operational)</li>
<li>Devboxes (Operational)</li>
<li>Western Europe (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Linux arm64 (Operational)</li>
<li>Artifacts (Operational)</li>
<li>Partner UI (Operational)</li>
<li>Linux arm64 on Apple Silicon (Operational)</li>
<li>Logging (Operational)</li>
<li>Dashboard (Operational)</li>
<li>Bazel (Operational)</li>
<li>Support (Operational)</li>
<li>Metrics (Operational)</li>
<li>HTTP Cache (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Northern Virginia, US (Operational)</li>
<li>Windows amd64 (Operational)</li>
<li>Gradle (Operational)</li>
 </ul>

Delayed webhook deliveries from GitHub leading to queue time

GitHub outage leading to job queuing

SSO login experiencing errors

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>Previews (Operational)</li>
<li>Devboxes (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Artifacts (Operational)</li>
 </ul>

Impact to multiple services

Status: Resolved All connections to the unhealthy instance were terminated, and error rates fully recovered. Affected components
 <ul>
 <li>Northern Virginia, US (Operational)</li>
<li>Dashboard (Operational)</li>
<li>GitHub Runners (Operational)</li>
<li>Linux arm64 (Operational)</li>
<li>Illinois, US (Operational)</li>
<li>Artifacts (Operational)</li>
<li>SAML SSO (Operational)</li>
<li>Support (Operational)</li>
<li>Linux arm64 on Apple Silicon (Operational)</li>
<li>Logging (Operational)</li>
<li>Metrics (Operational)</li>
<li>Devboxes (Operational)</li>
<li>Western Europe (Operational)</li>
<li>Remote Builders (Operational)</li>
<li>Linux amd64 (Operational)</li>
<li>Bazel (Operational)</li>
<li>MacOS arm64 (Operational)</li>
<li>Windows amd64 (Operational)</li>
<li>Gradle (Operational)</li>
<li>Social SSO (Operational)</li>
<li>Container Registry (Operational)</li>
<li>Previews (Operational)</li>
<li>Partner UI (Operational)</li>
<li>Turborepo (Operational)</li>
<li>HTTP Cache (Operational)</li>
 </ul>

Increased error rate for multiple services

Status: Resolved We're seeing recovery across the board, with runners no longer failing to acquire jobs. Queue time has returned to normal across all jobs.

Increased error rate from GitHub resulting in increased queue time

Status: Resolved We've applied a mitigation and are seeing full recovery. Affected components
 <ul>
 <li>Devboxes (Operational)</li>
<li>Container Registry (Operational)</li>
 </ul>

Is Namespace down?

Namespace status is UP

Lastest published events

A subset of devboxes is failing to start

Degraded registry and artifact storage performance

SAML SSO logins are failing

Namespace Services, Components and Regions

Recent Namespace events

A subset of devboxes is failing to start

Degraded registry and artifact storage performance

SAML SSO logins are failing

Delayed runner startup

Small subset of artifact and cache requests in Illinois region failing

Elevated instance creation time