We are closing this incident as we have not observed any further issues. However we are still awaiting an incident report from Zoom that we hope will provide more information around future mitigations.
Posted Feb 01, 2023 - 20:19 UTC
Monitoring
As of this morning, Zoom's infrastructure issues appear to have been resolved and our metrics show that the bot is successfully joining meetings in a timely fashion. We will continue to monitor for the day and we are awaiting confirmation from Zoom that the issue has been resolved.
Posted Jan 26, 2023 - 15:41 UTC
Identified
Zoom has reproduced the issue and is now working on a mitigation. At this time we do not have a timeline for resolution but we're following progress closely.
Posted Jan 26, 2023 - 05:57 UTC
Update
We are continuing to communicate with Zoom about the issue though no root cause has been discovered. We observed the issue to be mainly present during the range of 09:00-15:00 EST, so we are currently in a relatively quiet period.
Posted Jan 26, 2023 - 00:05 UTC
Update
Zoom has responded and is investigating the issue on their end.
Posted Jan 25, 2023 - 19:30 UTC
Update
Based on our observations, our recorder is experiencing high delays when connecting to the Zoom service (sometimes up to a few minutes). We've deployed a large increase in the recorder timeouts to mitigate the higher delays. While this doesn't solve the issue of the bot joining in a timely fashion, the bot should be more successful in eventually joining the meeting. We're continuing to investigate the root cause of the high delays in connecting to Zoom's meeting service.
Posted Jan 25, 2023 - 18:01 UTC
Update
The team is continuing to investigate the issue. We experienced a period of near total downtime which was resolved, but we are still observing elevated join failure rates due to timeouts from Zoom, we are continuing to investigate the issue.
Posted Jan 25, 2023 - 16:57 UTC
Update
The Grain Recorder continues to have issues joining meetings. The team is attempting various mitigations as the root cause is still unknown.
Posted Jan 25, 2023 - 15:10 UTC
Investigating
We have observed an increased rate of the Zoom bot timing out while attempting to join meetings. We are investigating the issue.