A/B Test Groups

What Are A/B Test Groups?

Test groups allow you to group two or more mailings to make head-to-head testing easier and more foolproof. Using a group helps protect against common testing mistakes by making sure the mailings' targeting criteria and rough size match, that the groups don't overlap, and that they're launched at the same time. They also help simplify the process by letting you review and launch the whole set of mailings from one screen, and use a few clicks to compare the mailings with statistical details and launch the winning mailing.

You can still run manual tests, using the Compare Mailings report to compare performance. In some cases this is the better approach. Test groups only make sense when mailings' targeting, size, and launch time all match: they're not designed for testing three petitions to three different list segments, or testing sending the same message on different days of the week. Groups also aren't available for petition delivery mailings or recurring mailings. And for small variations in message content, it may be simpler to create your own test using the mod filter.

Working With Test Groups

To start, click Create A/B Test Group on the Mailings dashboard. Enter a name; click drafts you'd like to include or, if you'd like ActionKit to create blank drafts for you, enter how many you want.

When you edit the targeting of any draft in the group, it will change targeting for all of them. When you pick a user limit, note you're limiting how many users each Mailing in the test can hit--it's not an overall limit for all mailings combined.

After you've created your test and entered final content, then click Proof and Send Test Group from any group member's Proof and Send screen (or click "Launch Test" in the drafts list). You'll see basic information on each draft and previews; if you see you need to make edits, click the title of any draft to go back to it. You can click "Edit test group settings" at the top of the page to add or remove drafts, change the title, or cancel the test. (You can access the same settings from the bottom of the targeting screen of any Mailing in the group.)

When everything's ready, scroll down to the bottom of the targeting screen. The system will check for obvious problems (e.g query not built, missing content, etc.) and hide the Send button if it finds any. When ready, click Send (or schedule, if applicable), take a moment for artistic appreciation of the haiku, and confirm. Even if you set up an immediate send, the test drafts will show in the Scheduled section on the Mailings Tab, sometimes for up to ten minutes.

Click the Test Results link on the dashboard to see the Compare Mailings report comparing the drafts. In the "results summary" view, you can see whether any draft is significantly ahead of the others (indicated by other drafts' numbers appearing greyed out) and see 95% confidence intervals for opens, clicks, and actions. (You might have to make a decision when without a single significant winner sometimes, but it's good to be aware of the uncertainty.)

Clicking one of the buttons in the Select Winner column of the report will load a copy of the winning Mailing, which excludes all of the test mailings and removes any limit on number of users. You can also click on a subject to see a mailing's individual report, or go back to the content-review screen you launched the test from. From an individual mailing report, you can click on a subject to choose a winning subject and winning mailing at the same time.

Those are the steps to running a test with test groups.

Other information you might find useful:

In test groups, some things work differently from usual, to keep mailings matching each other. Again, editing one Mailing's targeting changes targeting for the other mailings in the group, too. Test mailings' targets also won't overlap each other regardless of whether you use auto excludes, and will go to similar groups even if you use a non-random ordering like ZIP Code.

If a test Mailing stops early because of an error or is killed, you can reload it, and the reloaded mailing will assume the original mailing's test group targeting. Things work a little differently in this mode: you can't add or remove drafts from the test, and changes to the reloaded mailing's targeting won't (of course) affect the other sending drafts.

Internally, ActionKit targets test-group mailings using the MySQL MOD operator. Each group uses an arbitrarily-picked prime number, and each mailing in the test gets a WHERE condition like WHERE user_id MOD [prime] MOD [number of drafts in test] = [number of this draft]. You can see a particular draft's WHERE condition in its targeting summary. It's usually simpler to do reporting using mailing IDs, but you can also use the MOD expression directly for analysis.