[Resolved] Outage on Baseten embedding models

Baseten embedding models down

TemplateHighP95Latency

[Resolved] Palmyra vision is having issues when serviced through public api requests.

Palmyra Vision down through public api requests

[Resolved] Envoy pod q7fjq tripped its default ext_authz circuit breaker (max_connections: 1,024) after absorbing 57% of traffic due to a rollout without readiness probes, then getting hit by a GKE node removal reconnection storm. The circuit breaker locked for 17 minutes, returning HTTP 500 for ~62% of auth requests on that pod. 9.4% of all user-facing requests failed (5,203 of 55,449). The system self-recovered when the auth connection latency tail cleared naturally. The other two envoy pods were completely healthy throughout. A skynet-frontend restart at 17:41:15 was coincidental (old pods still alive when CB cleared at 17:42:00).

Writer Down

[Resolved] On 2026-03-10, Writer Agent experienced a SEV1 full outage driven by database lock contention during deployment. An in-band schema migration waited on long-lived idle-in-transaction sessions, causing query queueing on core.threads and rapid connection pool exhaustion across pods. Service recovered after terminating blocking sessions and stabilizing rollout; follow-up actions are in progress to harden migration safety and transaction lifecycle controls.

Writer Agent SEV1 Outage (DB Lock Contention During Migration)

[Resolved] In WA after authenticating and selecting the connector to retrieve context some users but not all are prompted to configure the connector in a loop until it fails to content.

Repeated Connector Configuration Prompts

[Resolved] The issue affecting WRITER Agent has been resolved. All systems are now operating normally. We apologize for any inconvenience this may have caused.

WRITER Agent stuck in "thinking"

[Resolved] The issue affecting our API services has been resolved. All systems are now operating normally. We apologize for any inconvenience this may have caused.

Is Writer down?

Writer status is UNKNOWN

Lastest published events

Baseten embedding models down

TemplateHighP95Latency

Palmyra Vision down through public api requests

Writer Services, Components and Regions

Recent Writer events

Baseten embedding models down

TemplateHighP95Latency

Palmyra Vision down through public api requests

Writer Down

Writer Agent SEV1 Outage (DB Lock Contention During Migration)

Repeated Connector Configuration Prompts