Status: Resolved A fix was rolled out to the replica provisioning path. Subsequent on-demand replica creations completed successfully. We are conducting a follow-up review to prevent recurrence.

We apologize for the disruption and any inconvenience caused. Affected components
 <ul>
 <li>Copy (Operational)</li>
<li>Copy (Operational)</li>
 </ul>

On-demand copies are failing

Status: Resolved We have successfully identified the root cause of the US East (AWS) public cluster disruption and fully restored both read and write operations; our team is currently processing the backlog of queued inserts accumulated during the incident.

Reads and writes degraded on US East (AWS) public cluster

Status: Resolved The incident is now resolved. Affected components
 <ul>
 <li>GCS Connector (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>Login (Operational)</li>
<li>Copy (Operational)</li>
<li>Events API (Operational)</li>
<li>Kafka (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>Query API (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Sink (Operational)</li>
 </ul>

Degradation on GCP europe-west2 public cluster

Status: Resolved The incident is now resolved. Affected components
 <ul>
 <li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
 </ul>

Errors when creating and deleting replicas

Status: Resolved We've confirmed that service has been restored fully. Affected components
 <ul>
 <li>DynamoDB Connector (Operational)</li>
<li>Events API (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>Sinks (Operational)</li>
<li>Query API (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Kafka (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>Kafka (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Events API (Operational)</li>
<li>Copy (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Sinks (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Query API (Operational)</li>
<li>Events API (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>DynamoDB Connector (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Kafka (Operational)</li>
<li>Kafka (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>BI Connector (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Sinks (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>Events API (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>Copy (Operational)</li>
<li>Login (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>Sinks (Operational)</li>
<li>Login (Operational)</li>
<li>Copy (Operational)</li>
<li>Events API (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>Events API (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Query API (Operational)</li>
<li>Query API (Operational)</li>
<li>Login (Operational)</li>
<li>Login (Operational)</li>
<li>Events API (Operational)</li>
<li>BI Connector (Operational)</li>
<li>Kafka (Operational)</li>
<li>Query API (Operational)</li>
<li>Sink (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>Sinks (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Query API (Operational)</li>
<li>Kafka (Operational)</li>
<li>Login (Operational)</li>
<li>Kafka (Operational)</li>
<li>Login (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Login (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>DynamoDB Connector (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Events API (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Query API (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>Copy (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>Copy (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Query API (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>Copy (Operational)</li>
<li>Sinks (Operational)</li>
<li>Kafka (Operational)</li>
<li>ClickHouse HTTP Interface (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Sinks (Operational)</li>
<li>Login (Operational)</li>
<li>API Endpoints (Operational)</li>
 </ul>

A significant fraction of Clickhouse replicas marked as unhealthy

Status: Resolved Ingestion is up to date now Affected components
 <ul>
 <li>Kafka (Operational)</li>
<li>Kafka (Operational)</li>
<li>Kafka (Operational)</li>
<li>Kafka (Operational)</li>
<li>Kafka (Operational)</li>
<li>Kafka (Operational)</li>
 </ul>

Kafka connector is working well, ingestion is catching up

Status: Resolved The situation is now resolved. There shouldn't be more errors. Affected components
 <ul>
 <li>API Endpoints (Operational)</li>
<li>Login (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Events API (Operational)</li>
<li>Kafka (Operational)</li>
<li>Copy (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>Sinks (Operational)</li>
<li>Query API (Operational)</li>
 </ul>

High error rate on GCP europe-west-3

Status: Resolved All data has been recovered and the system health is back to normal Affected components
 <ul>
 <li>Events API (Operational)</li>
<li>Query API (Operational)</li>
<li>Copy (Operational)</li>
 </ul>

Degraded performance in the AWS us-east public cluster.

Status: Resolved We've deployed a fix to improve the stability of the platform. Sorry for the inconvenience Affected components
 <ul>
 <li>API Endpoints (Operational)</li>
<li>Query API (Operational)</li>
 </ul>

We are investigate a degradation in AWS US EAST public cluster

Status: Resolved Cluster operating normally now Affected components
 <ul>
 <li>API Endpoints (Operational)</li>
<li>Events API (Operational)</li>
<li>Kafka (Operational)</li>
<li>Copy (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>Sinks (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Query API (Operational)</li>
<li>S3 Connector (Operational)</li>
 </ul>

GCP EU Public-a cluster is saturated

Status: Resolved The platform is stable again. We are going to review which has been the impact of the incident and contain the customers affected by it. Affected components
 <ul>
 <li>Events API (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>Kafka (Operational)</li>
<li>Query API (Operational)</li>
<li>Copy (Operational)</li>
<li>Copy (Operational)</li>
<li>Sinks (Operational)</li>
<li>Events API (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Kafka (Operational)</li>
<li>Query API (Operational)</li>
<li>Copy (Operational)</li>
<li>API Endpoint (Operational)</li>
<li>BI Connector (Operational)</li>
<li>DynamoDB Connector (Operational)</li>
<li>Login (Operational)</li>
<li>Login (Operational)</li>
<li>Query API (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Events API (Operational)</li>
<li>CSV Imports (Operational)</li>
<li>Sinks (Operational)</li>
<li>Events API (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Copy (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>Sinks (Operational)</li>
<li>Login (Operational)</li>
<li>DynamoDB Connector (Operational)</li>
<li>Sink (Operational)</li>
<li>Copy (Operational)</li>
<li>Kafka (Operational)</li>
<li>Kafka (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>Query API (Operational)</li>
<li>Login (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>Login (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>Events API (Operational)</li>
<li>Query API (Operational)</li>
<li>Kafka (Operational)</li>
<li>Copy (Operational)</li>
<li>Sinks (Operational)</li>
<li>Kafka (Operational)</li>
<li>Sinks (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>Query API (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>S3 Connector (Operational)</li>
<li>GCS Connector (Operational)</li>
<li>API Endpoints (Operational)</li>
<li>Events API (Operational)</li>
<li>Login (Operational)</li>
<li>GCS Connector (Operational)</li>
 </ul>

Is Tinybird down?

Tinybird status is UP

Lastest published events

On-demand copies are failing

Reads and writes degraded on US East (AWS) public cluster

Degradation on GCP europe-west2 public cluster

Tinybird Services, Components and Regions

Recent Tinybird events

On-demand copies are failing

Reads and writes degraded on US East (AWS) public cluster

Degradation on GCP europe-west2 public cluster

Errors when creating and deleting replicas

A significant fraction of Clickhouse replicas marked as unhealthy

Kafka connector is working well, ingestion is catching up