When testing a change in button color, you’d track click-to-open rates, which are more accurate than simple click rates, to see which color option triggers greater activity.
In our example, red was the winner over blue. If you wanted to see how a green button performed versus a red button, you’d have to conduct two separate tests in addition to the original above (for a total of three).
- a test of the control group (red) versus green
- a test of the winner of the red/green
For more accurate results, I’d suggest repeating the same tests several times. [Note: all tests in this blog post are hypothetical, so I’ve not provided results.]
For marketers who need to demonstrate ROI, A/B testing clearly has limitations since it can only be used to test for one variable at a time. If we kept our example very simple and emailed only one campaign a month, it would take three months to complete the test! If the deployment schedule is once a week, test results still wouldn’t be ready for four weeks.
A/B testing of more than one item can also introduce unknown variables which could throw your results out the window. How do you know if your winner was actually the result of your variable (button color, etc), and not just the content of your email? Was the day and time you sent the email a factor?
But what if you want to test five other colors? Or three different CTA’s? What if you have separate editorial images to go with each CTA color? Clearly, testing all these options one by one would not only be mind-numbing to set up, but take ages to run.
Are you getting dizzy yet? Data scientists do this for a living. They’d repeat the test over and over, and aggregate the data for statistically accurate results.
You shouldn’t need an advanced degree to conduct your own tests and improve your ROI. Enter our hero: multivariate testing.
Multivariate testing: study many versions at once
Don’t let the name put you off. If you break “multivariate” down into two parts, its meaning becomes clear: multi =many and variate = versions, a.k.a. it enables you to test multiple things at the same time.
Say someone on the marketing team wants to change the CTA text from “click here” to “buy now” but another person thinks it should actually be “learn more”. Sure! No problem. Your boss wants to try different colors for the button while you’re at it. Why not? What about changing the button placement too? Bring it on!!
We can test all of that at once – AND get accurate results! We’ll also be able to point EXACTLY to the one change that made the most difference.
Say you wanted to test images and button color. The three test colors are red, green and blue. The image choices consist of a cat, a dog and a bird. The control version has a cat image and red button that says “click here”. The other combinations to test are pictured: