A/B Testing: Gaining Insight Through Dividing and Analyzing

April 18, 2012

If you want more results from your direct mail or other marketing efforts, you need to continually refine your campaigns over time. Test new graphics. Test new messaging. Test new offers. That means A/B testing.

While A/B testing sounds complicated, it’s not. It’s splitting the mailing into two groups, modifying one variable at a time, and seeing which performs best. By continually testing one variable against another, you can identify those elements that consistently tend to produce the best results.

Case Study: Postcard Testing

For example, a company with a customer database of 5000 people might decide to create a postcard campaign with a discount code.

It might design the postcard, then modify the call to action to test the effectiveness of two different sets of text. To 2,500 people, it mails the postcard that says, “Offer ends this Tuesday! Use code A1.” To another 2,500 people, it sends the same postcard but with a different call to action that says, “Limited time offer! Use code A2.”

The company then tracks which group, A1 or A2, has the highest number of people redeem the offer. Say the postcard with code A1 has a 5% response rate (125 of the 2,500 people in that group used the code to buy a product) and the postcard using the code B1 has a 3% response rate (75 of the recipients used the code to buy a product). This tells the company which call to action is the most effective with its target audience. Most likely, it will use that same call to action in the future.

You can use this approach to test every element of your campaign. Use it to test graphics. Background colors. Starbursts and drop dates. The more you test, the more you can refine your campaigns for peak effectiveness.

Case Study: Newsletter Testing

One healthcare provider sending patient newsletters did A/B testing over a period of years. It assigned each article a unique tracking code paired to the household receiving the newsletter. As those households were tracked through the patient database, the healthcare provider was able to see which articles had the most impact.

The first year, the provider’s ROI was $62 for every dollar spent on the newsletter. The next year, that rose to $78. The next year, it rose to $88. Clearly, tracking was worth the effort!

How to Choose the Split

When it comes to choosing the split, you can divide the campaign evenly or select out a smaller portion to use as a control group. The key is to select names and addresses randomly so that each split is a mirror image of the other and includes enough names so that the results are statistically significant. Then pay attention to what you learn and use it to knock your next campaign out of the park.

Need help? Talk to us about what A/B splits make the most sense for your next campaign.

The key to A/B testing is to select names and addresses randomly so that each split is a mirror image of the other and includes enough names so that the results are statistically significant.

TO THE TOP