Status: Resolved This incident is now resolved. Between approximately 12:00 UTC on 3 October and 07:30 UTC on 5 October, customers in Europe (eu-west-1) experienced intermittent failures when pushing NuGet packages. 

We have applied changes to the affected services and confirmed that package pushes have returned to normal.


Pulling and downloading packages was unaffected throughout, as were all other package formats. Affected components
 <ul>
 <li>Package Uploads (Operational)</li>
 </ul>

Intermittent Issue pushing NuGet packages

Status: Resolved This incident is now resolved.

We have confirmed that the API has returned to normal operations. Affected components
 <ul>
 <li>REST API (api.cloudsmith.io) (Operational)</li>
 </ul>

Service Degradation On Our Web API

Status: Resolved We have resolved the issue with pulling Docker images Affected components
 <ul>
 <li>REST API (api.cloudsmith.io) (Operational)</li>
 </ul>

Issues Pulling Docker Images

Status: Resolved We have now release a fix to resolve this issue. Our OIDC service is operating normally Affected components
 <ul>
 <li>OpenID Connect (Operational)</li>
 </ul>

Increase in latency for OIDC requests

Status: Resolved The root cause was identified, and the issue is now resolved. Affected components
 <ul>
 <li>Package Sync (Operational)</li>
 </ul>

Package synchronization delays

Status: Resolved The root cause was identified and the issue is now resolved. Affected components
 <ul>
 <li>Block Until Scanned (Early Access) (Operational)</li>
 </ul>

Policies with upstream tags are not blocking packages when Block Until Scanned is enabled.

Status: Resolved OSV advisory backlog processing has completed, and returned to normal processing at 07:00 PM UTC. All systems are now operating normally. Affected components
 <ul>
 <li>Continuous Risk Detection(OSV) (Operational)</li>
<li>Policy Management (Operational)</li>
 </ul>

OSV vulnerability data delays.

Status: Resolved This incident is now resolved. Between 19:16 and 20:11 UTC, customers experienced request latency and timeouts affecting package downloads and API requests in the US East region. We have confirmed that these services have now returned to normal operations. Affected components
 <ul>
 <li>REST API (api.cloudsmith.io) (Operational)</li>
<li>Package Downloads (Operational)</li>
 </ul>

Request Latency and Timeout in US region

Status: Resolved We have confirmed that all services are operating as normal. Affected components
 <ul>
 <li>Continuous Risk Detection(OSV) (Operational)</li>
<li>Policy Management (Operational)</li>
 </ul>

Delay in Policy Evaluations

Status: Resolved We have confirmed that our services have returned to normal operations. Affected components
 <ul>
 <li>REST API (api.cloudsmith.io) (Operational)</li>
<li>Package Downloads (Operational)</li>
 </ul>

Request latency and timeout

Status: Resolved OSV advisory processing resumed at 23:50 UTC on August 25, and advisory freshness returned to normal at 04:35 UTC on August 26. All systems are now operating normally. Affected components
 <ul>
 <li>Continuous Risk Detection(OSV) (Operational)</li>
 </ul>

Delay in advisory data published on osv.dev

Status: Resolved The issue has now been resolved, and package caching to the repositories has resumed.
Packages pulled between 17:00 and 18:45 UTC will be missing from the repositories. These missing packages will be cached during subsequent pulls.


Note: Package caching to the repositories was impacted. Package pulls remained unaffected throughout the incident. Webhooks subscribed to the Package Downloaded event were also impacted during this period, and missed events will not be replayed. Affected components
 <ul>
 <li>Upstreams (Operational)</li>
 </ul>

New upstream packages may not appear in repositories

Status: Resolved Scope has been determined to be non-production only, so the incident is marked as non-impacting. Affected components
 <ul>
 <li>Cloudsmith Web App (app.cloudsmith.com) (Operational)</li>
 </ul>

Intermittent UI timeouts affecting the Package view

Status: Resolved No further impact to API or Web applications outside of the 1 minute window Affected components
 <ul>
 <li>Cloudsmith Web App (app.cloudsmith.com) (Operational)</li>
<li>REST API (api.cloudsmith.io) (Operational)</li>
<li>Legacy Web App (cloudsmith.io) (Operational)</li>
 </ul>

Errors affecting API and web requests

Status: Resolved We have confirmed that our work has returned services to normal operations. Affected components
 <ul>
 <li>Package Downloads (Operational)</li>
 </ul>

Download Request Latency for Python Packages

Status: Resolved We've reverted a change and confirmed resolution. 

This did not impact Enterprise (custom) domains or Enterprise Customers with custom domains. Affected components
 <ul>
 <li>Package Uploads (Operational)</li>
<li>Package Downloads (Operational)</li>
 </ul>

Intermittent authentication failures affecting Docker and Helm OCI logins in all regions

Status: Resolved Packages have successfully resynced and package syncing is operating as expected Affected components
 <ul>
 <li>Package Sync (Operational)</li>
 </ul>

Intermittent RPM package upload sync failures affecting package synchronization

Status: Resolved We have confirmed that our change has returned stability to the service Affected components
 <ul>
 <li>Package Downloads (Operational)</li>
 </ul>

Status: Resolved Services have become fully operational and the backfill for the impacted SBOMs have completed. We are continuing to monitor the platform to ensure stability. Affected components
 <ul>
 <li>Package Sync (Operational)</li>
<li>Security Scanning (Operational)</li>
 </ul>

Intermittent failures when syncing Docker images.

Status: Resolved Services have recovered and we are continuing to monitor the platform to ensure stability. Affected components
 <ul>
 <li>REST API (api.cloudsmith.io) (Operational)</li>
<li>Package Downloads (Operational)</li>
 </ul>

Minor service degradation in Europe region

Status: Resolved System has returned to normal operational levels. Affected components
 <ul>
 <li>Cloudsmith Web App (app.cloudsmith.com) (Operational)</li>
<li>REST API (api.cloudsmith.io) (Operational)</li>
<li>Legacy Web App (cloudsmith.io) (Operational)</li>
 </ul>

Partial outage impacting normal operations globally

Status: Resolved All systems are operational. Affected components
 <ul>
 <li>REST API (api.cloudsmith.io) (Operational)</li>
<li>Package Downloads (Operational)</li>
 </ul>

Elevated API and package download latency in Europe and US East regions

Status: Resolved The mitigation steps taken have resolved the issue. Latency has returned to normal levels, and we will continue monitoring for stability. Affected components
 <ul>
 <li>REST API (api.cloudsmith.io) (Operational)</li>
<li>Package Downloads (Operational)</li>
 </ul>

Is Cloudsmith down?

Cloudsmith status is UP

Lastest published events

Intermittent Issue pushing NuGet packages

Service Degradation On Our Web API

Issues Pulling Docker Images

Cloudsmith Services, Components and Regions

Recent Cloudsmith events

Intermittent Issue pushing NuGet packages

Service Degradation On Our Web API

Issues Pulling Docker Images

Increase in latency for OIDC requests

Package synchronization delays

Policies with upstream tags are not blocking packages when Block Until Scanned is enabled.