Failure to start recordings

Incident Report for Grain

Resolved

Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.
Posted Mar 24, 2026 - 19:58 UTC

Update

The team rolled back to the backup region due to continued higher than normal error rates.
Posted Mar 24, 2026 - 18:31 UTC

Monitoring

We've moved back to the main region and we're seeing new bots successfully be launched. Some issues are lingering around double scheduled bots and we are making adjustments to clean those up.
Posted Mar 24, 2026 - 17:46 UTC

Update

We are seeing signs that the main region is coming back online. Local captures should start capturing correctly.
Posted Mar 24, 2026 - 17:02 UTC

Update

The team has moved to a new region for the upstream provider and new bot recordings are starting to succeed.
Posted Mar 24, 2026 - 15:40 UTC

Investigating

An outage with an upstream provider is causing failures for new recordings (both bot and local capture). The team is investigating potential mitigations.
Posted Mar 24, 2026 - 15:32 UTC
This incident affected: Grain Recorder.