AB Testing capability

[marek-andrsk]

We’d like to have an ability to run experiments in Trezor Suite, using Feature Flags.

Experiment Configuration

Experiments would use Message System to config Feature Flags (i.e. there will be no UI to config experiments, deploy of the messaging system will be required - independed from Suite release though).

https://github.com/trezor/trezor-suite/blob/develop/docs/features/message-system.md

This needs to be extended to allow configuration:

• Experiment ID (e.g. rename-trade-section)
• Multiple Variation IDs (e.g. control, buy-swap) - this may be unlimited, but in reality we won’t use more than 4 for one experiment
• Each would have distribution percentage set between 0 - 100%

Traffic distribution

Distribution would be based on mathematical function applied to instanceID.

• number will be generated from instanceID
• we need to be able to edit test configuration and change the % - ideally this should only extend / reduce existing pool of test segment accounts, not randomize it completely

Tracking, etc.

Anything else - tracking, logic, will be implemented in the Suite as part of each experiment/variation. The reporting will use IDs from the Message System config.

[tomasklim]

Experiments would use Message System to configure Feature Flags (i.e. there will be no UI to config experiments, deploy of the messaging system will be required - independent from Suite release though).
Experiment ID (e.g. rename-trade-section)
Multiple Variation IDs (e.g. control, buy-swap) - this may be unlimited, but in reality, we won’t use more than 4 for one experiment
Each would have a distribution percentage set between 0 - 100%

[STew790]

QA OK
More info can be found here https://www.notion.so/satoshilabs/AB-Testing-on-mobile-18adc526060680ccb64cf4ddad4f780e