Skip to content

Conversation

@mrchocha
Copy link

@mrchocha mrchocha commented Aug 13, 2025

📋 Summary

This PR introduces a QA Benchmark Seed containing a set of intentional, toggleable bugs for use in QA benchmarking and performance evaluation. These bugs are safe, isolated, and can be enabled/disabled locally or via environment variables without impacting production.


🐛 Included Demo Bugs

Bug Name Trigger Flag Scope Description
Shift+Enter Sends Message localStorage.QA_BUG_SHIFT_ENTER_SEND = '1' Client Pressing Shift+Enter sends message instead of inserting newline
Muted Channel Still Notifies QA_BUG_MUTED_CHANNEL_NOTIF=1 Server Muted channel still triggers desktop notifications
Draft Lost on Room Switch localStorage.QA_BUG_ERASE_DRAFT = '1' Client Draft disappears when switching rooms quickly
Paste Image Clears Text localStorage.QA_BUG_PASTE_IMAGE_CLEARS_TEXT = '1' Client Pasting image clears existing composer text
Reaction Count Mismatch localStorage.QA_BUG_REACTION_COUNT_MISMATCH = '1' Client Reaction count doesn’t update properly after removal
Thread Counter Not Decremented localStorage.QA_BUG_THREAD_COUNTER_NOT_DECREMENTED = '1' Client Thread counter remains >0 after deleting last reply

🔧 Usage

Each bug is documented with:

  • Enable/Disable instructions

  • Reproduction steps

  • Modified files

  • Expected incorrect behavior

These can be found in qa-benchmark/ along with tickets.csv, which lists the bugs as realistic QA tickets.


🛡️ Safety

  • Bugs are opt-in only (disabled by default)

  • No side effects without explicitly enabling the respective flags

  • No changes to production behavior unless deliberately toggled


@mrchocha mrchocha changed the title feat(benchmarking): added bugs to QA benchmarking feat(benchmarking): added bugs to QA benchmarking [FOU-80] Aug 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants