Today we’ll be showing you a new way to A/B test. While this method has existed for years, it is a bit of a hack and isn’t common knowledge. This A/B testing method uses random sampling, and is just one of the many examples showing the vast capabilities of Marketo.
Random Sampling is a work-around using smart campaigns and static lists. It should be used in nested programs in engagement streams. This method allows you to incorporate A/B testing in engagement programs, where you wouldn’t be able to do otherwise.
Below are the flow steps in a smart campaign which will allow you to group program members into specific test audiences. You can create the random sample test audiences for all emails in just one smart campaign.
You can add as many test audiences as needed. If you’d like, you can create up to 100 test emails for one email send, each with a 1% random sample to make sure each list makes up for 1% of the total test audience. Keep in mind that you’ll need to create as many lists as test emails you create. This can take some time.
If using this method, you can’t select a winner, as these emails will always be sent out to the full audience. In a nurture program, the emails will be sent out at whatever cadence you have set in the streams.
Even though you can keep your members in one specific list throughout the entire program, we recommend creating new sample lists for each email send. You can also create additional lists to divide up list Week 1A by Sales and Marketing. Feel free to play around and get creative with the smart campaign logic.
What can this test?
Anything! Since this test is done manually, you can test anything you’d like, just remember you’ll have to create multiple emails.
You’ll be testing whole emails, but if you clone the emails and only change certain features such as subject line, or from names, you can test just these specific variables.
How is a winner declared?
There is no way to declare a winner in Marketo, however you can simply change the smart campaign in each email program to send everyone the best email.
To find out which email performed the best use the Email Performance Report to compare the results.
Pros and Cons
Pros:
- Allows you to maintain a defined A and B audience throughout all emails, though we recommend a random sample for each email
- You can test as many emails as you want
Cons:
- Takes more time to setup
- Marketo won’t provide metrics dashboard from this test
