Whereby - Some users are experiencing longer than expected connection times – Incident details

All systems operational

Some users are experiencing longer than expected connection times

Resolved
Degraded performance
Started almost 2 years agoLasted about 22 hours

Affected

Meeting rooms

Degraded performance from 4:18 PM to 2:01 PM

Embedded API

Degraded performance from 4:18 PM to 2:01 PM

Updates
  • Update
    Update

    The team has compiled a complete post-mortem of the incident, which is available for viewing at the linked document.

  • Resolved
    Resolved

    After monitoring overnight, we are resolving this incident.

    Cause

    The run time for one of our scheduled job’s became longer than it was scheduled for, which caused a spike in one of our databases load. This led to slow performance of that specific database and caused the performance degradation.

    Prevention Steps

    • We have improved that scheduled job’s performance
    • We have implemented new alarms to be early notified in case of future similar occurrences and be able to solve the issue without any customer impact.
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result. Based on early results connection times and recording/streaming events should be working as expected.

    UPDATE: We will continue monitoring until Wednesday 14 June 2023, but all systems should be operational and working as expected.

  • Investigating
    Investigating

    We are currently investigating longer connection times and failed join events to rooms. Cloud recording and live-streaming may also fail to start or take longer to start.