How You Should Not Do A/B Testing

A/B testing might seem like the Infinity Gauntlet of CRO but it needs to be snapped thoughtfully.

Sure, it works great to test variations of landing pages.

However, in other cases, it's not so straightforward.

‍

Ex:

• When you've just launched a site, jumping straight into A/B tests is pretty useless as traffic is so minuscule
• Setting the wrong goal

If you're testing a keyword based search vs faceted search, then number of searches as an engagement indicator is a poor choice.

Why?

Since application of each facet yields a new search result, it will naturally create more searches, which doesn't necessarily translate to user value.

• Testing obvious things

If you just cut down a form from 7 fields to 4 fields or used a much simpler captcha, you don't need to invest time to test these variations.

Simply apply & move on.

• The Downtown effect

If there's only one road going from a suburb to downtown, then residents will have no choice but to take it, regardless of how bumpy it is.

Similarly, say your marketing SaaS has a core flow for "Send Email" & you're testing a couple of variants to see which yields more email sends.

Both will convert similarly since paying users need to get their job done even if they hate 1 flow.