Status: Resolved Summary

Problem: We observed a delay in processing status updates for GitHub checks due to a backlog in our event processing system.

Impact: A small number of pull requests (PRs) experienced delays in status check updates. While the majority of the system remained functional, a specific segment of the processing queue was affected, causing a lag for those specific updates over several hours.

Cause: The delay was caused by a high volume of retries generated by unrecoverable permission errors on a limited number of requests, which blocked a portion of our processing capacity. This occurred during a period of high event volume, which led to the database reaching maximum capacity and slowing down the overall processing workflow.

Resolution: The issue has been resolved. We scaled up our database capacity, increased the number of processing units to handle the higher volume, and implemented logic to stop retrying updates that fail due to permanent permission errors. All backlogs have been cleared, and system performance has returned to normal for all pull requests.

Key Timeline (IDT)

• August 21, 2026, 06:30 IDT: An incident was declared as delays in status updates for a subset of requests became visible.
• August 21, 2026, 06:50 IDT: A public status page was published to inform customers of the degradation.
• August 21, 2026, 07:15 IDT: The database was scaled up to address processing bottlenecks.
• August 21, 2026, 07:45 IDT: Processing capacity was doubled by increasing the number of active queue partitions.
• August 21, 2026, 08:10 IDT: A corrective update was deployed to stop unnecessary retries on unrecoverable permission errors.
• August 21, 2026, 08:45 IDT: The event backlog was fully drained, and processing returned to real-time.
• August 21, 2026, 09:00 IDT: The incident was officially resolved.

Root Cause

The incident was caused by a combination of high event volume and a lack of "fail-fast" logic for permanent errors. Specific requests with incorrect permission settings generated a massive volume of retry attempts. Because the system treated these permanent permission errors as temporary, it repeatedly retried them, which blocked a specific processing partition and prevented updates for a small number of pull requests from being processed. This was further exacerbated by the database reaching its resource limits, creating a temporary slowdown in the update workflow.

Actions Taken

• Scaled Database Resources: Increased the capacity of the primary database to handle the increased load and improve processing speed.
• Increased Processing Parallelism: Doubled the number of active partitions in the event queue to ensure all available processing units were utilized.
• Deployed Retry Exclusion Logic: Updated the system to identify unrecoverable permission errors and stop retrying them, preventing queue blockages. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

Scan Performance Degredation

Status: Resolved 1. Summary

On August 19, 2026, some customers experienced slower page loading and data display, as well as delays when starting or completing pull request scans. 

The disruption was caused by a managed memory database entering a repeated restart cycle and not recovering automatically. This created a temporary slowdown in processing. Expected service behavior was restored after corrective actions, including replacement and increased capacity for the affected service. The provider is continuing to investigate why the service was unable to return to normal operation automatically.

2. Key Timeline (IDT)

• August 19, 2026, 16:10 IDT: We identified performance degradation affecting parts of the platform and pull request scanning.
• August 19, 2026, 16:30 IDT: An initial corrective change was applied and service behavior was monitored.
• August 19, 2026, 16:46 IDT: We confirmed that some delays were continuing and expanded the investigation.
• August 19, 2026, 17:14 IDT: We began working with our service provider to investigate instability in the managed memory database.
• August 19, 2026, 18:46 IDT: Recovery actions were in progress, with temporary mitigations in place to reduce customer impact.
• August 19, 2026, 19:06 IDT: The managed memory database capacity update completed, and platform responsiveness and pull request scanning returned to expected behavior.

3. Root Cause

The managed memory database entered a repeated restart cycle and was unable to recover automatically. This caused delays in the processing systems that support platform responsiveness and pull request scans.

The affected service underwent a routine replacement process, followed by a capacity increase during the recovery effort. While the capacity increase completed successfully and restored expected behavior, AWS is still researching why the service did not come back up normally after the restart cycle. Their detailed root-cause analysis is pending.

4. Actions Taken

• Applied corrective updates to reduce immediate processing demand.
• Temporarily adjusted the status update process to reduce reliance on the affected service.
• Restarted affected processing components to restore responsiveness.
• Worked with the service provider to replace affected capacity and complete a capacity increase.
• Monitored platform performance and scan processing until expected behavior was confirmed.

4b. Action Items

• Review the service provider’s detailed root-cause analysis when available.
• Improve monitoring to detect similar recovery delays earlier.
• Add safeguards to reduce the impact of temporary processing slowdowns.
• Introduce separate Redis capacity for each service to isolate workloads and reduce the impact of an issue in one service on others. Affected components
 <ul>
 <li>Application/UI (US Environment) (Operational)</li>
 </ul>

Platform and PR scans slowness

Status: Resolved Some GitHub pull request status checks in the EU region were delayed by up to 30 minutes before completing. 

No checks were lost and no action is needed - delayed checks completed automatically. We identified the cause, and fixed it. We're monitoring closely. Affected components
 <ul>
 <li>API (EU Environment) (Operational)</li>
 </ul>

Delayed GitHub PR status checks

Status: Resolved GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

GitHub incident can affect Cycode

Status: Resolved Summary

PR secret scans experienced processing lag, causing delays in processing pull requests.

The issue was caused by timeouts when calling the model validation service. The model validation service was experiencing elevated latency and errors because compute nodes were scheduled on a subnet that had exhausted its available IPs.

This slowed PR secret scan processing and caused consumer lag to build up.

To resolve the issue, we temporarily disabled the feature that produces new PR secret scans so the queue could drain. We also resolved the underlying infrastructure issue by moving nodes to subnets with available free IPs.

Customer Impact

Customers experienced delayed PR secret scans.

Other scan types remained operational.

Root Cause

The secret scanner calls the AI model validation service to classify detections during PR scans.

The model validation call had a 30-second timeout configured. When the model validation service became slow or unresponsive, each failed request could block a scan consumer for the full 30 seconds before timing out.

The underlying issue was in the model-serving infrastructure. One of the private subnets had run out of available IPs, and new nodes were created in that exhausted subnet, leaving model-serving pods in a pending state.

This resulted in elevated latency and errors in the model validation service and reduced the processing capacity of PR secret scans.

Contributing Factors

• Model validation timeouts. The model validation service timeout was 30 seconds. When the service was degraded, scan consumers waited for the full timeout before the request failed, which created a processing bottleneck.
• No fallback to bypass AI validation for PR scans. There was no mechanism to skip AI model validation specifically for PR secret scans while allowing the rest of the scan to continue. The available kill switch stopped producing PR secret scans entirely. This meant the mitigation required a tradeoff between allowing processing lag to continue or temporarily stopping PR secret scans.
• Customer communication gap. Internal alerts for PR scan lag did fire, but the status page was not updated promptly to reflect the degradation. As a result, there was a period in which we had internal visibility into the issue while customers did not have the same visibility through the status page.

Mitigation and Recovery

A pre-existing feature flag was used to stop new PR secret scans from being produced.

This stopped new messages from being added to the PR secret scan queue and allowed the existing backlog to drain.

The backlog was cleared shortly after the mitigation was enabled.

In parallel, the underlying model-serving infrastructure issue was resolved by moving nodes to subnets with available free IPs, restoring model-serving capacity.

Once the infrastructure issue was resolved and the backlog had drained, PR secret scanning was restored.

Learnings and Corrective Actions

• Review the AI model validation timeout. The 30-second model validation timeout meant that when the model-serving layer degraded, each scan consumer could remain blocked for the full timeout before failing. We are reviewing the timeout and retry behavior. This work must also account for the risk that shorter timeouts and faster retries could increase request volume against an already degraded model-serving service.
• Add a graceful degradation path for AI validation. The system currently has a kill switch for PR secret scanning, but does not have a graceful degradation mechanism for the AI validation step. We are working on a mechanism that can disable AI model calls specifically for PR scans, allowing scans to continue while avoiding the dependency on a degraded model-serving layer. The goal is to allow scans to continue with reduced functionality rather than stopping PR secret scanning entirely.
• Improve alerting and incident communication. The alerting infrastructure detected PR scan lag, but the status page was not updated promptly. We are improving the process for escalating service degradation and communicating customer-facing impact through the status page.
• Improve model-serving infrastructure resilience. The underlying infrastructure issue occurred because nodes were scheduled on a subnet with no available IPs. The immediate issue was resolved by moving nodes to subnets with free IPs. We are continuing to improve the resilience of the model-serving infrastructure to reduce the likelihood that infrastructure capacity issues can affect PR scan processing.

Conclusion

The incident was caused by degradation in the AI model validation service due to an infrastructure issue where nodes were scheduled on a subnet with exhausted IP capacity.

The resulting model validation timeouts slowed PR secret scan processing and caused consumer lag to build.

We mitigated the incident by temporarily stopping new PR secret scans, allowing the backlog to drain, and restoring model-serving capacity by moving nodes to subnets with available free IPs.

We are following up with improvements to model-serving resilience, timeout and retry behavior, runtime configuration, graceful degradation, alerting, and customer communication. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

Degraded performance in Secrets Pull Request scanning

Status: Resolved GitHub component: Git Operations, Webhooks, Pull Requests Original GitHub incident: https://stspg.io/f0cgtdc0kqxg

Status: Resolved GitHub component: Pull Requests Original GitHub incident: https://stspg.io/ssd9z8l2g46v

Status: Resolved GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

Status: Resolved Root Cause

The incident was caused by a deployment of one service that ran a database index creation. The team has identified an issue with the way we perform index creations as database migrations. 

During a deployment the pods with newest image of the service attempted to create an index on a big table. The team has identified that the index creation took 7 minutes. However, during index creation pods were not responsive, and as a result, Kubernetes deemed them as unhealthy pods and attempted to retry those pods after 5 minutes.

As a result, because the pod got killed before the index creation was fully completed, the database transaction was rolled back. Then, subsequent pods attempted to create the index again, dying after 5 minutes.

This lead to the database being in unhealthy state, and the service was down. 

The team has rolled back the deployment, and killed all replicas that attempted to create the index. Thanks to that, the service and the database was in healthy state again.

 

Why safety measures did not help

Cycode provides a safety mechanism that unblocks all Pull Request scans after a specific period of time, giving each scan a maximum duration before the Pull Request is unblocked. However, because the service that is responsible for triggering and completing scans, as well as this safety net, was down, the process couldn't behave as expected. We acknowledge this gap and are working on strengthening this area of our system.

 

Action items

• The team is actively investigating enhancements and new safety protocols that can be put in place in order to have another safety net preventing Pull Request scans being stuck in case of any incident.
• The team is investigating changes to the index creation process. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

We have noticed degraded performance in scanning

Status: Resolved The platform is now fully operational and processing normally Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
<li>Application/UI (US Environment) (Operational)</li>
 </ul>

We’re investigating an issue causing older events to be reprocessed

Status: Complete Maintenance has completed Affected components
 <ul>
 <li>Application/UI (US Environment) (Under maintenance)</li>
<li>API (US Environment) (Under maintenance)</li>
 </ul>

Maintenance: Infrastructure upgrade

Status: Resolved Root Cause

The incident was caused by a deployment of one service that ran a database migration. The team has identified that this migration contained faulty code and as a result lead to database overload when attempting to deploy the service. As a result, the service was partially down until the deployment was reverted. During this time, all scans were processed with lower than expected performance. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

Degraded performance in scans

Status: Resolved GitHub component: API Requests Original GitHub incident: https://stspg.io/vr201n49yl53

Status: Complete Complete Affected components
 <ul>
 <li>Application/UI (EU Environment) (Operational)</li>
<li>API (EU Environment) (Operational)</li>
 </ul>

Status: Resolved The issue has been resolved, and we are continuing to monitor the situation. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

Degraded performance in IaC Pull Request scans

Status: Resolved GitHub component: Pull Requests Original GitHub incident: https://stspg.io/sm1tp7kfm4vj

Status: Resolved GitHub component: API Requests, Pull Requests Original GitHub incident: https://stspg.io/j5c80shxqm53

Status: Resolved GitHub component: Webhooks, Pull Requests Original GitHub incident: https://stspg.io/syhr80rth84z

Status: Resolved Functionality is fully restored; we are continuing to monitor to ensure everything remains stable. Affected components
 <ul>
 <li>Application/UI (US Environment) (Operational)</li>
<li>API (US Environment) (Operational)</li>
 </ul>

Inaccurate violation counts in some dashboards and panels.

Issues with Violations and Custom Dashboards (Prod-US)

Status: Resolved The incident has been resolved, and the platform is operating normally. Affected components
 <ul>
 <li>Application/UI (US Environment) (Operational)</li>
 </ul>

Platform Slowness

Status: Resolved A fix has been applied and functionality is fully restored; we are continuing to monitor to ensure everything remains stable. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
<li>API (EU Environment) (Operational)</li>
 </ul>

Cycode CLI 3.17.1/2 Secrets scans failures

Status: Resolved We’re continuing to monitor detection processing and the associated delays in violation status updates (including auto-resolution) Affected components
 <ul>
 <li>Application/UI (EU Environment) (Operational)</li>
 </ul>

Delays in violations status updates in EU

Status: Resolved Summary
On June 10, 2026, customers may have experienced slowness in SAST Pull Request scanning. The root cause has been identified in insufficient processing capacity in one of the services used in SAST Pull Request scans flow. This resulted in degraded performance and a portion of scans taking longer time than usual to start and to complete. The processing capacity has since been increased and adequate alerts have been created to ensure the issue does not occur again.

Key Timeline (IDT)

• June 10, 2026, 17:50 IDT: Performance degradation began.
• June 10, 2026, 18:15 IDT: The team noticed degraded performance and began working on stabilizing the system.
• June 10, 2026, 18:40 IDT: The system was confirmed to be fully operational.


Root Cause
The incident was caused by an insufficient processing capacity in one of the services involved in SAST Pull Request scan flow. This means, that under heavy load, the service struggled to maintain its operational velocity. The team also found that the SAST Pull Request scans flow was not fully optimized and it lead to over-utilization of the service.

Actions Taken

• Increased processing capacity of the bottleneck service.
• Created alert to detect bottlenecks quicker.
• Optimized behavior of the service.


Action Items

• Further optimize SAST Pull Request scanning flow. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

Degraded performance in SAST PR scans

Status: Resolved The system is now fully operational. There should be no more degraded performance.

 

Summary
We observed a period of slowness and intermittent timeouts affecting various system functions in the EU region, including the application interface and pull request (PR) scans. The issue was primarily caused by a processing system reaching its network and memory capacity limits, exacerbated by a high volume of automated activity from a single source. We have since upgraded the underlying infrastructure and implemented safeguards to prevent similar high-volume activity from impacting the system. The issue is now fully resolved, and all services have returned to expected performance levels.

Key Timeline (IDT)

• July 13, 2026, 11:44 IDT: Incident detected following reports of UI slowness and PR scan delays.
• July 13, 2026, 12:19 IDT: Infrastructure bottleneck identified; decision made to upgrade the processing cluster.
• July 13, 2026, 12:26 IDT: A high-volume automated process was identified and disabled to reduce immediate load.
• July 13, 2026, 13:09 IDT: Infrastructure upgrade completed; network throughput returned to normal levels.
• July 13, 2026, 15:35 IDT: All backlogs cleared, and the incident was officially resolved.

Root Cause
The incident was triggered by a combination of factors: a processing cluster reached its maximum network bandwidth and memory capacity due to an undersized configuration for the current workload. This was further strained by a specific automated workflow that generated an unusually high volume of update requests. Additionally, a configuration difference in the message processing pipeline in the EU region prevented the system from effectively handling the resulting backlog.

Actions Taken

• Upgraded Infrastructure: The processing cluster was upgraded to a higher-capacity instance type to provide more network bandwidth and memory.
• Disabled High-Volume Source: A specific client identifier responsible for excessive traffic was temporarily disabled to restore system stability.
• Restored Connectivity: Affected service components were restarted to ensure they re-established clean connections to the upgraded infrastructure.
• Increased Processing Parallelism: The number of partitions in the affected message queue was increased to allow the system to process the backlog more quickly.

Action Items

• Enhance Monitoring: Implement new alerts for network and memory utilization to detect capacity issues before they impact customers.
• Optimize Update Workflow: Refactor the status update process to batch requests, significantly reducing the load on the processing system.
• Implement Rate Limiting: Introduce safeguards to prevent any single source from consuming disproportionate system resources.
• Standardize Regional Configurations: Conduct an audit to ensure infrastructure and message queue settings are consistent across all regions. Affected components
 <ul>
 <li>API (EU Environment) (Operational)</li>
<li>Application/UI (EU Environment) (Operational)</li>
 </ul>

Degraded performance

Status: Resolved The issue affecting Maestro AI services has been resolved. Maestro Risk Explorability, Risk AI Remediation, Maestro Remediation, and Graph AI are now available and operating normally.

 

Summary

On July 9, 2026, customers using the Maestro service in the European production environment experienced a period of service unavailability. The issue began following a configuration update that inadvertently changed the service's regional routing. This caused the system to attempt connections through a network path that lacked the necessary permissions and to a region where specific processing models were unavailable. The issue has been fully resolved, and service has been restored to all affected customers.

Key Timeline (IDT)

• July 9, 2026, 12:02 IDT: The incident was identified and an investigation was initiated.
• July 9, 2026, 12:07 IDT: Public notification was issued regarding the service interruption.
• July 9, 2026, 13:00 IDT: A network configuration fix was applied, restoring primary connectivity.
• July 9, 2026, 13:39 IDT: Service was fully restored after implementing model fallbacks, and the incident was marked as resolved.

Root Cause

The service interruption was triggered by a recent update to the authentication and configuration process. This update introduced a conflict in how the system identified its operating region. Specifically, an automated update process overrode manual settings, routing traffic to a different regional endpoint. This new path was blocked by a missing network security rule and attempted to use a processing model that was not supported in that specific region, leading to service failure.

Actions Taken

• Restored Network Connectivity: Manually updated network security rules to allow secure traffic through the new regional endpoint.
• Implemented Model Fallbacks: Configured the system to use alternative processing models to ensure immediate service availability while long-term regional configurations were adjusted.
• Updated Status Communications: Maintained real-time updates for stakeholders and customers throughout the recovery process.

Action Items

• Standardize Configuration Precedence: Update the deployment workflow to prevent automated processes from silently overriding critical environment settings.
• Infrastructure Audit: Conduct a comprehensive review of network security rules across all regions to ensure consistency and prevent similar connectivity gaps.
• Enhance Automated Monitoring: Implement end-to-end health checks and synthetic probes to detect regional connectivity issues automatically before they impact users.
• Improve Deployment Policies: Establish new guidelines to ensure that configuration changes are deployed and validated in production-like environments more frequently to reduce the risk of "stale" updates. Affected components
 <ul>
 <li>API (EU Environment) (Operational)</li>
 </ul>

Maestro AI service disruption

Status: Resolved AWS has confirmed that the issue has been fully mitigated and we are currently not observing any related issues. Affected components
 <ul>
 <li>Application/UI (US Environment) (Operational)</li>
<li>API (US Environment) (Operational)</li>
 </ul>

Infrastructure Provisioning Delays

Status: Resolved Summary

During the incident, customers experienced significant delays and temporary disruptions across SAST, SCA, CCA, and Secret repository scans and push events. The issue was caused by a surge in reachability scanner jobs that overwhelmed the processing queue, compounded by scanner pods requesting excessive CPU and memory, infrastructure resource limits being reached, and inefficiencies in job prioritization and retry logic. As a result, processing capacity was improperly consumed and a large job backlog accumulated. A series of corrective updates were deployed to stabilize the environment, and the processing environment has since returned to expected performance levels.

Impact

Customers experienced delays for SAST, SCA, CCA, and Secret repository scans and push events, with some requests delayed by several hours and a peak queue size of over 64,000 jobs. Lower priority scans such as Trivy, Syft, and CCA were most affected, though high-priority jobs were eventually processed without further delay.

Key Timeline (IDT)

• 21.06.2026, 17:16 IDT: A surge in reachability scanner jobs caused the CycodeX queue to grow rapidly.
• 22.06.2026, 10:07 IDT: The issue was identified by an on-call engineer.
• 22.06.2026, 12:55 IDT: We increased the scanning platform resources to process more jobs.
• 22.06.2026, 14:10 IDT: A fix that lowered new reachability scanners was deployed to production.
• 22.06.2026, 18:43 IDT: Existing reachability scanners' priority was lowered.
• 23.06.2026, 09:12 IDT: Scans with higher priority were processed. Only lower priority scans remained, including CCA.
• 23.06.2026, 13:51 IDT: A fix that reduced communication overload to Kubernetes was deployed. The scanning platform started processing scan jobs much faster.
• 23.06.2026, 17:34 IDT: The queue was fully processed.

Root Cause

The issue was triggered by a combination of factors:

1. Reachability scanner job surge -- A surge in reachability scanner jobs caused the CycodeX queue to grow rapidly, which led to resource bottlenecks in the cluster and a peak queue size of over 64,000 jobs.
2. Excessive pod resource requests -- Due to configuration bugs, scanner pods requested excessive CPU and memory, which prevented efficient scheduling and amplified the resource bottlenecks in the cluster.
3. Infrastructure resource limits -- AWS VPC subnet IP and EKS API limits were reached, restricting the cluster's ability to scale and schedule new work.
4. Job prioritization and retry inefficiencies -- Inefficiencies in job prioritization and retry logic meant lower priority scans (Trivy, Syft, CCA) competed for capacity and were most affected, while the backlog continued to grow.

Actions Taken

• Increased cluster and node pool capacity.
• Fixed job prioritization to deprioritize reachability scans.
• Capped resource requests for scanner pods to enable efficient scheduling.
• Deployed additional fixes to the scanning platform.
• Opened AWS support tickets to address resource limits.
• Restored monitoring and logging.
• Cleared the job backlog; the queue now processes new jobs as they arrive.

Action Items

• Improve monitoring to better understand the behavior of the processing environment.
• Improve scanning optimization and prioritization for all scan types. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

SAST, SCA and Secret scans slowness

Status: Resolved Summary
During the incident, customers experienced delays and temporary disruptions in on demand scans. The issue was caused by a number of redundant SCA scans as well as unoptimized reachability scans. As a result, processing capacity was improperly used for redundant, duplicate, scans. A corrective update for both issues was deployed to stabilize the environment, and the processing environment has since returned to expected performance levels.

Key Timeline (IDT)

• 01.06.2026: Unoptimized code responsible for reachability scans was introduced.
• 03.06.2026: A faulty code responsible for redundant SCA scans was introduced.
• 11.06.2026, 16:00 IDT: Increased processing delays were detected, and an investigation into the scanning service began.
• 11.06.2026, 17:00 IDT: The root cause for redundant SCA scans was identified, and the team began working on improving the behavior.
• 12.06.2026, 10:30 IDT: The root cause of redundant SCA scans was resolved.
• 12.06.2026, 17:00 IDT: The root cause for degraded performance in reachability scans was identified.
• 15.06.2026, 16:00 IDT: A corrective update was deployed to restore the optimization of reachability scans.
• 15.06.2026, 17:00 IDT: System stability was confirmed.


Root Cause
The issue was triggered by a combination of two issues:

1. Redundant SCA scans -- Upon receiving a new branch created event from the source code manager, the system attempted to scan all available branches that were configured to be scanned in the repository. In extreme cases, the team has found a magnitude of new scans started instead of one scan, when a branch was created. A faulty code introduced this bug on 03.06.2026 and got resolved on 12.06.2026
2. Unoptimized reachability scans -- Reachability scanner in the processing environment, has been configured to scan a set of programming languages. However, due to misconfiguration, the scanner was also attempting to scan programming languages that have not been configured to be scanned. Additionally, the scanner has been running on programming languages that are not stable yet, and in extreme cases, it caused a portion of scans to be stuck until they got timed out.


Actions Taken

• Deployed a corrective update to remove redundant SCA scans.
• Deployed a performance upgrade to reachability scans.


Action Items

• Improve monitoring to better understand the behavior of the processing environment.
• Improve scanning optimization for all scan types. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

Degraded performance in on demand scans

Status: Resolved Summary
During the incident, customers experienced delays and temporary disruptions in scans. The issue was caused by a degraded performance of a process responsible for starting the scans. This means that the processing environment was underutilized even though there were enqueued scans. A corrective update was deployed to stabilize the environment, and the system has since returned to expected performance levels.

Key Timeline (IDT)

• 07.06.2026, 07:00 IDT: Degraded performance of scans began.
• 08.06.2026, 11:00 IDT: Degraded performance was noted and the investigation began.
• 08.06.2026, 13:15 IDT: A bottleneck was identified and a corrective update was deployed. 
• 08.06.2026, 14:00 IDT: It was confirmed that the update increased the throughput of the system.
• 09.06.2026, 14:00 IDT: System stability was confirmed, and the system was restored to normal operation.


Root Cause
The issue was triggered by degraded performance of a process responsible for starting scans. This caused the processing environment to be underutilized (even as low as 30% processing capacity), despite there being enqueued scans.

The root cause of the degraded performance has been identified in a race condition when attempting to rerun a failed scan. When attempting to rescan, the system occasionally encountered an error, because the first attempt of a scan failed, and the job was in terminating state. This resulted in extensive retries of the starting the rescans, which caused the degraded performance.

Actions Taken

• Deployed a corrective update to the bottleneck process.
• Restored expected processing performance.
• Improved monitoring to detect degraded performance.


Action Items

• Further improve monitoring to detect degraded performance.
• Further improve the performance of the bottleneck process. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
 </ul>

Status: Resolved The issue was resolved. The system is fully stable now Affected components
 <ul>
 <li>Application/UI (US Environment) (Operational)</li>
<li>API (US Environment) (Operational)</li>
 </ul>

Slow SAST PR scans

Status: Resolved Summary

Between April 14 and April 28, 2026, a processing issue occurred within our scanning workflow that prevented a subset of code update events from being fully processed. As a result, some security findings may not have been updated or resolved as expected in the platform. The issue was caused by an update to the processing system that introduced a conflict in how data was written to the database. The issue has been fully resolved, and all affected data has been reprocessed to ensure accuracy.

Key Timeline (IDT)

• April 14, 2026, 09:00 IDT: A new update was deployed to the processing system which introduced the underlying issue.
• April 28, 2026, 15:30 IDT: A subsequent system update was rolled out that inadvertently corrected the processing error.
• May 4, 2026, 11:00 IDT: Internal teams identified reports of inconsistent status updates and began a technical investigation.
• May 6, 2026, 10:00 IDT: A formal incident was initiated to analyze the full scope of the impact and coordinate remediation.
• May 7, 2026, 14:00 IDT: Data collection was completed to identify all repositories and branches that required reprocessing.
• May 8, 2026, 08:00 IDT: Reprocessing of affected data began for several scanning modules.
• May 9, 2026, 18:00 IDT: Initial reprocessing for the majority of modules was successfully completed.
• June 11, 2026, 12:00 IDT: Final reprocessing for all remaining modules was completed, and the incident was officially closed.

Root Cause

The issue was triggered by a change in the update process that attempted to perform multiple database operations simultaneously within the same request. This led to connectivity conflicts and prevented certain background tasks from completing successfully. Consequently, a small portion of events were not recorded, leading to delays in updating the status of findings for customers.

Actions Taken

• Deployed a corrective update to the processing system to ensure stable database interactions.
• Conducted a comprehensive data analysis to identify all events that were missed during the affected period.
• Triggered system-wide rescans for all affected branches to restore the expected state of all findings.
• Verified that all processed updates are now reflecting correctly across the platform.

Action Items

• Enhance system logging to provide better visibility into background processing failures.
• Implement automated alerts for specific error patterns to decrease detection time.
• Improve the scalability of the underlying processing services to handle high volumes more efficiently.
• Develop improved internal tools for bulk data correction and status verification. Affected components
 <ul>
 <li>Application/UI (US Environment) (Operational)</li>
<li>EU Environment (Operational)</li>
<li>API (US Environment) (Operational)</li>
<li>Application/UI (EU Environment) (Operational)</li>
<li>API (EU Environment) (Operational)</li>
<li>US Environment (Operational)</li>
 </ul>

Unprocessed Push Event

Status: Resolved Summary
On June 10, 2026, customers may have experienced intermittent issues with their GitHub integrations due to an external authentication incident on GitHub's side. This resulted in some integrations temporarily reporting invalid access tokens and a brief pause in repository syncing and scanning. The issue was resolved once the external provider restored their authentication services. All integrations have automatically returned to normal operation, and no customer action is required.

Key Timeline (IDT)

• June 10, 2026, 18:20 IDT: The external provider began experiencing authentication issues affecting API requests.
• June 10, 2026, 18:20 IDT: Our processing system began receiving intermittent unauthorized responses, leading to temporary integration errors and paused syncing.
• June 10, 2026, 19:39 IDT: The external provider identified and mitigated the faulty component, resolving the incident.
• June 10, 2026, 19:39 IDT: Integrations automatically re-validated, and expected processing behavior was restored.



Root Cause
The incident was caused by an external authentication failure at GitHub. During this window, the provider's API erroneously returned "Unauthorized" responses for a portion of traffic. Our system interpreted these responses as revoked access tokens, which triggered temporary integration error states and paused background workflows.

Actions Taken

• Monitored the external incident status and its impact on our integration workflows.
• Verified that integrations automatically recovered and resumed syncing once the external provider restored service.
• Confirmed that no manual re-authentication was necessary for affected customers.



Action Items

• Improve internal handling of intermittent authentication errors to better distinguish between temporary external outages and actual token revocations.
• Enhance monitoring to more quickly identify when third-party service disruptions impact integration health. Affected components
 <ul>
 <li>Application/UI (US Environment) (Operational)</li>
 </ul>

GitHub Authentication Incident Affecting Cycode GitHub Integrations

Status: Resolved The incident has been resolved. We will provide the RCA shortly. Affected components
 <ul>
 <li>US Environment (Operational)</li>
<li>Application/UI (EU Environment) (Operational)</li>
<li>API (EU Environment) (Operational)</li>
<li>Application/UI (US Environment) (Operational)</li>
<li>API (US Environment) (Operational)</li>
<li>EU Environment (Operational)</li>
 </ul>

Slow SAST scans

Status: Resolved Summary

Between 16:00 and 17:30 UTC on May 28, we observed an issue where integration connections were intermittently reported as invalid. During this period, the processing of pull requests was paused until connection tokens were successfully refreshed. As a result, some pull requests may have experienced delays or remained in a stuck state. The issue is now resolved, and processing has resumed normally.

Key Timeline (UTC)

• May 28, 16:00 UTC: A high rate of integration connections were reported as invalid, and pull request processing was paused.
• May 28, 17:00 UTC: The external service began returning successful responses, and the system started marking integrations as valid.
• May 28, 17:30 UTC: Connection tokens were successfully refreshed and pull request processing resumed.

Root Cause

The issue was caused by an external service provider returning "no access" responses during a routine validity check. This caused the system to mark integrations as invalid and pause processing. The failures occurred simultaneously across many unrelated accounts and were resolved without any changes on our side, indicating a service disruption on the provider's side.

While the observed behavior clearly points to the external provider, we are continuing to work with GitHub to better understand the underlying cause of this transient failure.

Actions Taken

• Restored expected processing behavior once connection tokens were successfully refreshed.
• Monitored the system to ensure all integrations returned to a valid state.

Action Items

• Improve monitoring to detect similar external service anomalies.
• Add alerting on simultaneous validity-check failures across unrelated accounts. Affected components
 <ul>
 <li>Application/UI (US Environment) (Operational)</li>
<li>EU Environment (Operational)</li>
<li>US Environment (Operational)</li>
<li>API (US Environment) (Operational)</li>
<li>API (EU Environment) (Operational)</li>
<li>Application/UI (EU Environment) (Operational)</li>
 </ul>

GitHub PR Delays

Status: Resolved Summary
Between May 15 and May 28, 2026, a processing delay was identified affecting dependency update workflows for several large accounts. During this period, customers may have experienced significant lag in the time required to process and reflect dependency updates within the system. The issue was caused by an inefficiency in how the database retrieved specific project records, leading to temporary performance degradation. The issue has been fully resolved, and processing speeds have returned to expected levels.

Key Timeline (IDT)
May 17, 2026, 09:00 IDT: Initial reports of increased processing times for dependency updates were observed.
May 24, 2026, 14:30 IDT: System resources were increased to provide temporary relief while the investigation continued.
May 28, 2026, 11:15 IDT: Technical analysis identified a specific database query that was not utilizing the most efficient lookup path.
May 30, 2026, 16:45 IDT: A corrective update to the database indexing structure was deployed.
May 31, 2026, 10:00 IDT: Monitoring confirmed that processing latency returned to sub-millisecond levels and the backlog was cleared.

Root Cause
The root cause was a performance regression in the update workflow’s data retrieval process. The database’s automated query planner began selecting a less efficient path to locate project references. Because the system was searching through a very large volume of records, this inefficient path resulted in the system scanning thousands of unnecessary rows for each update. This led to high disk activity and significant delays, particularly for accounts with a high density of shared dependencies across many projects.

Actions Taken

1. Resource Scaling: Temporarily increased database memory and processing power to mitigate the immediate impact on customers.
2. Database Optimization: Performed a deep analysis of query execution plans to identify the specific bottleneck.
3. Index Refinement: Replaced an existing database index with an optimized version that ensures the query planner always selects the most direct path for data retrieval.
4. Validation: Verified the fix across all environments to ensure the new indexing structure supports both data integrity and high-speed lookups.

Action Items

1. Enhanced Monitoring: Implement more granular alerts for database query performance to detect similar planning inefficiencies before they impact customers.
2. Optimization Standards: Update internal database design guidelines to prevent overlapping index patterns that can confuse automated query planners.
3. Performance Testing: Incorporate “cold cache” scenarios into performance testing for large-scale data operations to better simulate real-world disk I/O constraints. Affected components
 <ul>
 <li>API (US Environment) (Operational)</li>
<li>API (EU Environment) (Operational)</li>
<li>EU Environment (Operational)</li>
<li>Application/UI (US Environment) (Operational)</li>
<li>Application/UI (EU Environment) (Operational)</li>
<li>US Environment (Operational)</li>
 </ul>

Is Cycode down?

Cycode status is UP

Lastest published events

Scan Performance Degredation

Platform and PR scans slowness

Delayed GitHub PR status checks

Cycode Services, Components and Regions

Recent Cycode events

Scan Performance Degredation

Platform and PR scans slowness

Delayed GitHub PR status checks

GitHub incident can affect Cycode

Degraded performance in Secrets Pull Request scanning

GitHub incident can affect Cycode