Feb 29, 10:35:55 GMT+0
Resolved -
We are noticing that cloud recordings end prematurely, roughly after 5 mins. We are deploying a potential fix..
Feb 29, 11:47:21 GMT+0
Resolved -
An update to the underlying OS/runtime of our recording service caused recordings to crash after a few minutes. This crash was sadly not detected by our automated nor manual tests, which are always run before such deployments. The reason why we did not pick this up was the error only presented itself after a certain time, which is outside the time we have encoded in for our test runs. We have now rolled back the change and the feature should be working as expected. We are yet to find the root cause of the issue, and have scheduled time to identify what went wrong, and also discuss how our test setup can be adjusted to catch similar regressions in the future..
Feb 22, 17:29:00 GMT+0
Resolved -
This incident has been resolved..
Feb 22, 08:35:00 GMT+0
Identified -
Due to some SFU crashes in Europe, users in this region might temporarily experience issues like black tiles or no audio from the other participants. We are currently working on a fix for the issue..
Feb 22, 14:18:46 GMT+0
Monitoring -
We have implemented a fix and we are now monitoring it. .
Feb 20, 17:55:45 GMT+0
Resolved -
This incident has been resolved..
Feb 20, 13:50:00 GMT+0
Monitoring -
Cloud recording might unexpectedly fail due to an on-going incident one of our providers is having. The issue has been identified, fixed and is currently monitored. .
Jan 26, 08:00:00 GMT+0
Resolved -
This morning we deployed some changes to our meeting experience which prevented some users accessing their rooms. After the first error report we rolled back our change and verified that users were able to access their rooms again.
We have yet identified what exactly went wrong, as this only affected a subset of our users.
Jan 26th 2024
----------
08:53 CET - Started deploy of new build which used script attribute crossorigin="anonymous" to allow us to get detailed insights into unhandled exceptions
08:59 CET - Deploy complete
09:18 CET - Got report from user not being able to access their room
09:21 CET - Started rollback to latest working version
09:29 CET - Rollback complete, verified the failing changes had been removed
So downtime (for some users) was roughly between 09:00 CET and 09:30 CET
.
Sep 6, 17:15:32 GMT+0
Resolved -
We've reverted some changes in our signup flow that were causing unexpected issues during the business signup flow. The trial signup process is now working as expected.
Aug 29, 08:15:00 GMT+0
Investigating -
Room Insights participant logs are impacted for all users that used screensharing in their sessions
.
Sep 4, 15:00:00 GMT+0
Resolved -
This issue has now been resolved, but the data during this period involving participant detail logs may be incomplete.
Billing, calculation of participant minutes and room level and room session level insights remained unaffected throughout this period. This issue only affected detailed participant data in the administration dashboard..
Jun 23, 18:45:03 GMT+0
Resolved -
From approximately 15:47-16:25 CEST our server partner in Sydney experienced outages that may have caused participants in that part of the world to experience delays in connecting to our platform..
Jun 13, 18:10:29 GMT+0
Monitoring -
We implemented a fix and are currently monitoring the result. Based on early results connection times and recording/streaming events should be working as expected.
UPDATE: We will continue monitoring until Wednesday 14 June 2023, but all systems should be operational and working as expected..
Jun 13, 16:18:17 GMT+0
Investigating -
We are currently investigating longer connection times and failed join events to rooms. Cloud recording and live-streaming may also fail to start or take longer to start..
Jun 14, 14:01:23 GMT+0
Resolved -
After monitoring overnight, we are resolving this incident.
## Cause
The run time for one of our scheduled job’s became longer than it was scheduled for, which caused a spike in one of our databases load. This led to slow performance of that specific database and caused the performance degradation.
## Prevention Steps
- We have improved that scheduled job’s performance
- We have implemented new alarms to be early notified in case of future similar occurrences and be able to solve the issue without any customer impact..
Jun 15, 20:15:21 GMT+0
Resolved -
The team has compiled a [complete post-mortem](https://docs.google.com/document/d/1Rjl2NnMdW7aPhnhEkBhE-5RID89PxKHcB-zNhxba-_U/edit?usp=sharing) of the incident, which is available for viewing at the linked document..
May 18, 12:28:37 GMT+0
Identified -
We're aware of an issue with delays in insights data updating via the dashboard and endpoint. The root cause has been identified and a temporary fix in place. There will be delays in data refreshing until a permanent fix is in place..
May 23, 14:36:14 GMT+0
Resolved -
This incident has been resolved and we've observed expected behavior in our insights updating accordingly!.
Mar 31, 12:00:00 GMT+0
Resolved -
Our developers discovered a bug with how our WebSockets were functioning. The mechanism we have in place to automatically reconnect a user during a temporary disconnect, was not working properly. This may have resulted in users needing to more often refresh their page.
A fix has been deployed and moving forward the issue should be resolved..
Mar 1, 13:41:06 GMT+0
Identified -
We've identified an issue with the process that updates the room insights and session insights accessible from the dashboard and GET endpoints. We're actively working on a fix, but you can expect delays in the datas availability.
Data found in the insights overview section of the dashboard is unaffected..
Mar 8, 13:41:00 GMT+0
Resolved -
We've updated our data pipeline to properly refresh data as expected and have observed consistent behavior. This issue has been resolved.
Oct 14, 15:47:09 GMT+0
Resolved -
On October 14 we received an increase in reports of black video frames in meetings. After investigating we linked this issue to a recent deploy, and rolled back the deploy. We are investigating what specifically caused the issue in that deploy, but the black video frames issue has been resolved after rolling back..
Sep 26, 12:57:52 GMT+0
Investigating -
We are currently investigating this incident..
Sep 26, 13:34:10 GMT+0
Investigating -
While we investigate the issue, we have paused updates to Customer Insights dashboards. Customer Insights dashboards will update to reflect new data after the investigation is complete. Thanks for your patience..
Sep 26, 14:50:27 GMT+0
Monitoring -
After investigation, we've implemented a fix that we believe will resolve the issue. We are monitoring the fix and will provided further informationn.
Sep 27, 12:50:57 GMT+0
Resolved -
We’ve verified our fixes implemented yesterday successfully corrected the issue. We increased the capacity of our customer insights system to meet growing demand.
Sep 2, 19:37:00 GMT+0
Investigating -
Due to unexpected server downtime, users connecting to rooms were experiencing an error..
Sep 2, 20:07:00 GMT+0
Resolved -
The issue has been identified and resolved, all systems are operational..
Aug 15, 22:09:22 GMT+0
Investigating -
We are currently investigating this incident..
Aug 16, 00:08:58 GMT+0
Monitoring -
Due to a connection issue with a server partner in the Asian Pacific region, there were issues loading Whereby’s website. We've implemented a workaround, however some customers may experience longer load times when accessing rooms, until the root cause is addressed..
Aug 16, 12:25:12 GMT+0
Resolved -
After monitoring and continued improves, the issue has been resolved and normal functionality restored..
Jun 14, 18:33:56 GMT+0
Investigating -
We are currently investigating this incident..
Jun 14, 19:30:28 GMT+0
Identified -
We are actively working on a fix for this incident..
Jun 14, 19:45:05 GMT+0
Resolved -
Due to a faulty code deploy, our cloud recording service was temporarily unavailable for some customers. We’ve updated our infrastructure accordingly and service has now resumed.
Apr 29, 12:24:06 GMT+0
Investigating -
We are currently investigating this incident..
Apr 29, 13:25:02 GMT+0
Monitoring -
We've implemented a fix and are currently monitoring the result..
Apr 29, 19:00:08 GMT+0
Resolved -
After continued monitoring, we've verified that cloud recording functionality is fully restored and the issue resolved..
Apr 14, 15:38:29 GMT+0
Investigating -
Customers using our legacy embed component host URL may be experiencing issues. We are currently investigating this incident..
Apr 14, 15:49:09 GMT+0
Monitoring -
The issue has been identified and implemented a fix. We are currently monitoring the result..
Apr 14, 16:26:32 GMT+0
Resolved -
Due to a file hosting issue, the embed component legacy host source was temporarily unavailable. The file has been restored and the issue resolved..
Mar 8, 17:35:31 GMT+0
Investigating -
We’re investigating issues with upgrades to our Pro plan. We detected this issue today (March 8) at 09:00 Central Europe time, and will continue to update the status pending our investigation.
This is just a billing management issue only so your meetings will not be impacted. .
Mar 8, 18:12:48 GMT+0
Identified -
We were able to identify the issue based on a change that was made yesterday. Currently working on a fix. .
Mar 8, 19:15:43 GMT+0
Monitoring -
We implemented a fix and currently monitoring the result..
Mar 8, 20:17:37 GMT+0
Resolved -
We just resolved the issue!.
Mar 4, 15:05:00 GMT+0
Investigating -
The embed component (https://whereby.dev/embed/) is unavailable..
Mar 4, 15:25:00 GMT+0
Resolved -
The issue is now solved..
Feb 16, 14:20:00 GMT+0
Resolved -
The issue is now solved..
Feb 16, 13:30:00 GMT+0
Identified -
The webhook feature is disrupted and no events are being sent to endpoints..
Feb 15, 09:57:00 GMT+0
Investigating -
Webhooks for "room.session.ended" event types are not being sent..
Feb 16, 16:59:00 GMT+0
Investigating -
We are currently investigating this incident..
Feb 16, 19:30:00 GMT+0
Monitoring -
Webhooks that hadn't been sent are gradually being sent now..
Feb 16, 21:30:00 GMT+0
Resolved -
All delayed webhooks have been sent..
Oct 19, 14:39:48 GMT+0
Resolved -
The issue is now solved..
Oct 19, 13:06:00 GMT+0
Investigating -
Joining meeting rooms across our different products is slow. We are actively investigating the issue and monitoring the situation..
Oct 19, 14:00:00 GMT+0
Monitoring -
The issue is now resolved and we're monitoring our internal metrics to make sure that users aren't experiencing any more delays..
Sep 9, 07:49:00 GMT+0
Investigating -
We are currently investigating this incident..
Sep 9, 09:38:00 GMT+0
Resolved -
This issue is now resolved..
Jul 21, 11:12:00 GMT+0
Identified -
We have identified the issue, which is caused by a recent deployment to our web application..
Jul 21, 11:21:00 GMT+0
Monitoring -
We have successfully rolled back the faulty deployment and are monitoring that users can now use an integration as expected..
Jul 21, 11:31:00 GMT+0
Resolved -
We have confirmation that the issue is now resolved..
Jul 21, 10:49:00 GMT+0
Investigating -
We are investigating reports of an issue that causes participants to be automatically disconnected from rooms with an error whenever one of the participants uses an integration (including screen sharing)..