Some join links are unable to access the room

Summary

• Released Webinar Room component code changes caused regression and partial outage for a specific group of users

Impact

• Attendees couldn’t join some automated events properly.
• Only scheduled automated events with private chat preferences were affected. All other events worked fine.

Root Cause Analysis

• The engineering team worked on fixing one bug and unintentionally introduced a code change that caused the given incident.
• The QA team didn’t test the new code change properly and missed a new bug in the updated code. The end-to-end automated tests didn’t detect the system behavior change in the Staging environment.
• The updated code was deployed to Production, which caused regression.

Resolution and Recovery

• The engineering team identified the root cause of the regression and applied a new hotfix that solved the problem.

Actions Points

• We will improve our QA end-to-end automatic testing script to cover more combinations of event settings.
• We will improve our manual QA testing process to avoid missing such cases.
• We will continue improving our codebase to make it more reliable and fault-tolerant to newly introduced changes.