<p> Incident began at <strong>2026-02-27 04:37</strong> and ended at <strong>2026-02-27 06:35</strong> <span>(all times are <strong>US/Pacific</strong>).</span></p><div class="cBIRi14aVDP__status-update-text"><h1>Incident Report</h1>
<h2>Summary</h2>
<p>On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, Vertex Gemini API and Dialogflow CX in US regions and the global endpoint. The issue persisted for a duration of 1 hour and 58 minutes.</p>
<p>This is not the level of quality and reliability we strive to offer you, and we have taken immediate steps to improve the platform’s performance and availability.</p>
<h2>Root Cause</h2>
<p>This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. For some specific requests, this created code paths that eventually led to service disruptions and capacity loss for the safety filtering service. Consequently, customers encountered overload (429 and 503) errors for their queries, with some users reporting elevated error rates for specific models in US regions.</p>
<h2>Remediation and Prevention</h2>
<p>Google engineers were alerted to the issue via our automated monitoring system on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation.</p>
<p>Engineers identified the faulty configuration change and initiated a rollback to restore the previous stable configuration. Engineers also added more capacity to the service to stabilize it. Full service restoration was confirmed by 06:35 US/Pacific as the rollback propagated and servers became healthy. <br>
<br>
Google is committed to preventing a repeat of this issue and is taking the following actions:</p>
<ul>
<li>Reinforcing rollout processes to include mandatory validation checkpoints.</li>
<li>Improving alerting systems to monitor critical dependencies more closely.</li>
</ul>
<h2>## Detailed Description of Impact</h2>
<p>On Friday, 27 February 2026 between 04:37 and 06:35 US/Pacific, customers accessing Vertex Gemini APIs may have experienced the following:</p>
<ul>
<li><strong>Affected Models:</strong> All Vertex AI Gemini API models were affected, including gemini-2.5-flash, gemini-2.5-flash-lite, gemini-2.5-pro, gemini-3.0-flash-preview, gemini-3.0-pro-preview, gemini-2.0-flash, gemini-2.0-flash-lite.</li>
<li><strong>Error Experience:</strong>
<ul>
<li><strong>PayGo Customers:</strong> Experienced primarily 429 Resource Exhausted errors.</li>
<li><strong>Provisioned Throughput (PT) Customers:</strong> Received 503 Service Unavailable errors.</li>
<li>For PT customers, most errors stopped at <strong>06:00</strong>. For PayGo customers, most errors stopped at <strong>06:20</strong>.</li>
</ul>
</li>
<li><strong>Geographic Scope:</strong> Global endpoint, us-central1, us-east4, and other US regions were impacted.</li>
</ul>
</div><hr><p>Affected products: Agent Assist, Dialogflow CX, Google Cloud Support, Vertex Gemini API</p><p>Affected locations: Global, Montréal (northamerica-northeast1), São Paulo (southamerica-east1), Iowa (us-central1), South Carolina (us-east1), Northern Virginia (us-east4), Columbus (us-east5), Oregon (us-west1)</p>

RESOLVED: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

Iowa (us-central1)

South Carolina (us-east1)

Northern Virginia (us-east4)

Columbus (us-east5)

Oregon (us-west1)

Los Angeles (us-west2)

Salt Lake City (us-west3)

Las Vegas (us-west4)

Montréal (northamerica-northeast1)

Toronto (northamerica-northeast2)

São Paulo (southamerica-east1)

Santiago (southamerica-west1)

Belgium (europe-west1)

London (europe-west2)

Frankfurt (europe-west3)

Netherlands (europe-west4)

Zurich (europe-west6)

Milan (europe-west8)

Paris (europe-west9)

Turin (europe-west12)

Warsaw (europe-central2)

Finland (europe-north1)

Taiwan (asia-east1)

Hong Kong (asia-east2)

Tokyo (asia-northeast1)

Osaka (asia-northeast2)

Seoul (asia-northeast3)

Mumbai (asia-south1)

Delhi (asia-south2)

Singapore (asia-southeast1)

Jakarta (asia-southeast2)

Sydney (australia-southeast1)

Melbourne (australia-southeast2)

Access Context Manager

Agent Assist

AI Platform Prediction

AI Platform Training

Dallas (us-south1)

Mexico (northamerica-south1)

Berlin (europe-west10)

Madrid (europe-southwest1)

Bangkok (asia-southeast3)

Doha (me-central1)

Dammam (me-central2)

Tel Aviv (me-west1)

AlloyDB for PostgreSQL

Stockholm (europe-north2)

Anthos Service Mesh

API Gateway

Apigee

Apigee Hybrid

Application Integration

Artifact Registry

Assured Workloads

AutoML Natural Language

AutoML Tables

AutoML Translation

AutoML Video

AutoML Vision

Backup and DR

Backup for GKE

Bare Metal Solution

Batch

BigQuery Data Transfer Service

Binary Authorization

Certificate Authority Service

Certificate Manager

Cloud Armor

Cloud Build

Cloud Data Fusion

Cloud Data Loss Prevention

Cloud Domains

Cloud External Key Manager

Cloud Filestore

Cloud Firestore

Cloud HSM

Cloud IDS

Cloud Key Management Service

Cloud Load Balancing

Cloud Logging

Cloud Memorystore

Cloud NAT

Cloud Run

Cloud Shell

Cloud Spanner

Cloud Storage for Firebase

Cloud Vision

Cloud Workflows

Cloud Workstations

Colab Enterprise

Contact Center AI Platform

Contact Center Insights

Data Catalog

Database Migration Service

Dataform

Dataplex

Dataproc Metastore

Datastream

Developer Connect

Dialogflow CX

Document AI

Eventarc

GKE fleet management

Google App Engine

Google BigQuery

Google Cloud Bigtable

Google Cloud Composer

Google Cloud Dataflow

Google Cloud Dataproc

Google Cloud Datastore

Google Cloud Deploy

Google Cloud Functions

Google Cloud Identity-Aware Proxy

Google Cloud Migration Center

Google Cloud NetApp Volumes

Google Cloud Pub/Sub

Google Cloud Scheduler

Google Cloud SQL

Google Cloud Storage

Google Cloud Tasks

Google Compute Engine

Google Distributed Cloud Edge

Google Kubernetes Engine

Healthcare and Life Sciences

Hybrid Connectivity

Identity and Access Management

Infrastructure Manager

Integration Connectors

Key Access Justifications

Looker (Google Cloud core)

Managed Lustre

Managed Service for Apache Kafka

Managed Service for Microsoft Active Directory (AD)

Memorystore for Memcached

Memorystore for Redis

Memorystore for Redis Cluster

Memorystore for Valkey

Migrate to Virtual Machines

Model Armor

Network Connectivity Center

Network Security Integration

Parallelstore

Google Cloud

Is Google Cloud down?

Google Cloud status is UP

Lastest published events

RESOLVED: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

Google Cloud Services, Components and Regions

Recent Google Cloud events

RESOLVED: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

Get your FREE account now,
5 monitors included!

Is Google Cloud down?

Google Cloud status is UP

Lastest published events

RESOLVED: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

Google Cloud Services, Components and Regions

Recent Google Cloud events

RESOLVED: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!