Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A/B test enhancements #193

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions docs/channels/emails.rst
Original file line number Diff line number Diff line change
Expand Up @@ -478,3 +478,84 @@ Mautic Users can't unsubscribe and therefore the unsubscribe link looks like thi

Best practice is to create a Segment with a small number of Contacts to receive test Emails - for example, yourself - which ensures that you can fully test features such as unsubscribe behaviour.

Mautic A/B Testing: Sending Winner Variant to Contacts
=============================

Mautic's new feature allows users to run A/B testing with a segment of their contacts, determine the winning variant based on pre-configured criteria, and then send the winning email to the remaining contacts in the segment.

How it Works
--------------

Let's go through an example:

1. **Setting up the A/B Test**:

Assume you have a segment of 1,000 contacts. You create an A/B Test with three different emails (A, B, and C). You set the parent email to use 10% of all emails for the A/B tests, with a 3-hour delay to determine the winning email. The two children emails are each set to 33%.

The distribution may look like this:
- Email A: sent to 34 people
- Email B: sent to 33 people
- Email C: sent to 33 people

2. **Determining the Winning Criteria**:

For this test, let's say you've set the 'winning criteria' as the email open rate. The results come back as follows:
- Email A: Open rate is 25%
- Email B: Open rate is 15%
- Email C: Open rate is 10%

3. **Sending the Winning Email**:

After the pre-determined amount of time (3 hours in this case), the winning email (Email A) is sent to the remaining 70% of the segment using the standard `mautic:broadcast:send` command.

Commands
-----------

- `mautic: email:sendwinner` searches for all emails with variant settings for automatically sending the winning variant. This command can be used by a cron job.
- `mautic: email:sendwinner --id=101` works for a single email only.

Important Points
------------------

- To send the A/B test, the 'Send' button forces emails to be sent with publish dates as broadcast emails with `mautic:broadcast:send` command.
- Once a variant is chosen as the winner with command `mautic: email:sendwinner`, the other emails are set as unpublished, and the winning email becomes a regular email.
- The winning email is then sent as a regular email to the remaining contacts using `mautic:broadcast:send`.

Setting Up A/B Testing in Mautic Emails
--------------------------------------------

This guide will walk you through the process of enabling and setting up A/B tests for your emails in Mautic.

Enable and Setup A/B Tests for Email
---------------------------------------

During the process of creating an email, you will have the option to enable and set up A/B tests.

.. image:: images/ab-test1.png
:alt: Email A/B Testing Setup

Create Variant from Parent Email
----------------------------------

Once you have a parent email, you can create a variant from it. This variant will serve as an alternative to the original email for the A/B test.

.. image:: images/ab-test2.png
:alt: Creating Variant

Setup Traffic for Children Email
-----------------------------------

After creating a variant, you can set up the amount of traffic (or percentage of contacts) that each variant will be sent to.

.. image:: images/ab-test3.png
:alt: Setting Up Traffic

A/B Testing Summary
----------------------

After setting up your A/B test, you can view a summary of your setup. This summary includes information on the parent email, the variants, and the distribution of traffic between them.

.. image:: images/ab-test4.png
:alt: A/B Testing Summary

For further testing and adjustments, make use of Mautic's `mautic: email:sendwinner` command to send the winning variant, which can be used by a cron job for automatic sending, and make sure to regularly check the performance of your variants.
Binary file added docs/channels/images/ab-test1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/channels/images/ab-test2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/channels/images/ab-test3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/channels/images/ab-test4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.