Nelio A/B Testing Blog

Latest News and Updates in the A/B Testing World

pricingpage.png

The importance of A/A Testing (no, not a typo!)

Which version of the above pricing page you prefer? Can’t see the difference? Well, you’re right, there is none.

What you see is a good example of A/A Testing. In short, A/A Testing is an A/B Testing experiment where the two alternatives are in fact exactly the same.

Yes, I know, this looks kinda stupid. But we believe there are two good reasons to use A/A tests from time to time:

  • Checking your Split Testing tool. You’re relying on the results provided by your A/B testing tool (hopefully ours if you’re on WordPress) to make critical decisions about the look & feel, structure, navigation flow,… of your website. If the tool is not working as it should, it means that all these changes could be killing your conversion rate instead of improving it and make you lose lots of money. An A/A Test would help you to detect this. If the tool tells you that one variation in an A/A Test is significantly better than the other, well, it’s time to change
  • Reminder of the importance of a careful interpretation of the results. It may surprise you the results in an A/A test are not going to be exactly the same for both alternatives. Even if you’re showing exactly two clones of the same page to all your visitors, random effects will make that both clones have a slightly different conversion rate. Again, they can’t be significantly different but by no means they will be exactly equal. We’ve found this useful as a way to make our customers realize that they should not rush to select an alternative. As soon as a variation is slightly better we all feel the urge to stop the test (we don’t want to lose customers by showing the not-so-good alternative to them). Seeing that these rate differences also happen in an A/A test helps them to hold their horses in future tests. Remember, we may not like it but the underlying math in A/B Testing is there for a reason!

So, do you want to A/A Test us?

For more about WordPress and conversion optimization check us on twitter , facebook or Google+ or subscribe to our mailing list. And if you’re not yet on WordPress, let us help you migrate to WordPress.

Be sociable, share!

NelioThe importance of A/A Testing (no, not a typo!)
Share this post

1 comment

Join the conversation
  • Georgi - July 30, 2014 reply

    This is one of over 20 articles I’ve read on the topic and not understanding what the deal was with A/A testing (A/A/B, A/A/B/B etc.) I decided to share my thoughts on the issue from a statistical/hypothesis testing perspective. In general, I don’t think there is any merit in doing such tests. The results aren’t interesting cause they don’t tell us anything we don’t already know. My detailed response is here: http://blog.analytics-toolkit.com/2014/aa-aab-aabb-tests-cro/

    I see your point as far as “Reminder of the importance of a careful interpretation of the results” goes, but I would much prefer those people to read a bit more on sampling theory and statistical inference instead of relying on something shaky as an A/A test for that.

    Kind Regards,
    Georgi

Join the conversation