Resolved
Some users are experiencing longer than expected connection times

Started
June 13, 2023 at 4:18:17 PM
Status
Resolved after about 22 hours

Impact

Degraded performance
Affected
Embedded API
Meeting rooms
  • Resolved

    The team has compiled a complete post-mortem of the incident, which is available for viewing at the linked document.

  • Resolved

    After monitoring overnight, we are resolving this incident.

    Cause

    The run time for one of our scheduled job’s became longer than it was scheduled for, which caused a spike in one of our databases load. This led to slow performance of that specific database and caused the performance degradation.

    Prevention Steps

    • We have improved that scheduled job’s performance
    • We have implemented new alarms to be early notified in case of future similar occurrences and be able to solve the issue without any customer impact.
  • Monitoring

    We implemented a fix and are currently monitoring the result. Based on early results connection times and recording/streaming events should be working as expected.

    UPDATE: We will continue monitoring until Wednesday 14 June 2023, but all systems should be operational and working as expected.

  • Investigating

    We are currently investigating longer connection times and failed join events to rooms. Cloud recording and live-streaming may also fail to start or take longer to start.