Our recorder service began exhibiting intermittent network failures on individual servers, starting around 14:45 PDT following a rollout of a new release. Instead of resolving itself via our health checks, the failures slowly cascaded across the services, triggering our alarms. After attempting to diagnose and resolve the problem, the engineering team rolled back the release at 17:00 PDT. We are continuing to investigate the root cause of the issue.
The failures manifest as 1) new recordings failing to start and 2) currently running recordings cutting off before the end of the meeting.