Marketing Technology

Random Sampling: A “New” Way to A/B Test

Today we’ll be showing you a new way to A/B test. While this method has existed for years, it is a bit of a hack and isn’t common knowledge. This A/B testing method uses random sampling, and is just one of the many examples showing the vast capabilities of Marketo.

Random Sampling is a work-around using smart campaigns and static lists. It should be used in nested programs in engagement streams. This method allows you to incorporate A/B testing in engagement programs, where you wouldn’t be able to do otherwise.

Below are the flow steps in a smart campaign which will allow you to group program members into specific test audiences. You can create the random sample test audiences for all emails in just one smart campaign.

You can add as many test audiences as needed. If you’d like, you can create up to 100 test emails for one email send, each with a 1% random sample to make sure each list makes up for 1% of the total test audience. Keep in mind that you’ll need to create as many lists as test emails you create. This can take some time.

If using this method, you can’t select a winner, as these emails will always be sent out to the full audience. In a nurture program, the emails will be sent out at whatever cadence you have set in the streams.

Even though you can keep your members in one specific list throughout the entire program,  we recommend creating new sample lists for each email send. You can also create additional lists to divide up list Week 1A by Sales and Marketing. Feel free to play around and get creative with the smart campaign logic.

What can this test?

Anything! Since this test is done manually, you can test anything you’d like, just remember you’ll have to create multiple emails.

You’ll be testing whole emails, but if you clone the emails and only change certain features such as subject line, or from names, you can test just these specific variables. 

How is a winner declared?

There is no way to declare a winner in Marketo, however you can simply change the smart campaign in each email program to send everyone the best email.

To find out which email performed the best use the Email Performance Report to compare the results. 

Pros and Cons

Pros:

  • Allows you to maintain a defined A and B audience throughout all emails, though we recommend a random sample for each email
  • You can test as many emails as you want

Cons:

  • Takes more time to setup
  • Marketo won’t provide metrics dashboard from this test

Jack Laushway

Jack is a Marketo Certified Associate who is passionate about delivering success to clients and supporting their marketing efforts through creativity and innovation.

You can share this article on:

The tricky part? Harnessing it.

We’ll send you one email a week with content you actually want to read, curated by the DS team.

related

You might also enjoy

Marketing Technology

Two methods of implementing compliance into your Marketo smart campaigns.

Marketing Technology

Modular offering provides resources to onboard, execute, and measure Marketo programs.

Marketing Technology

Find out what companies are asking prospective MaaS providers.