The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
Posted Oct 20, 2025 - 18:03 UTC
Update
We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
Posted Oct 20, 2025 - 16:07 UTC
Update
AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
Posted Oct 20, 2025 - 14:55 UTC
Update
We are continuing to monitor for any further issues.
Posted Oct 20, 2025 - 13:59 UTC
Monitoring
AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
Posted Oct 20, 2025 - 13:09 UTC
This incident affected: Recording Processing and Grain Recorder.