Most email A/B-tests mean nothing
False positives, wrong conclusions, and the framework you should use
An A/B-test analysis
“Let’s A/B-test this” is a very serious thing to say.
It evokes big words. Hypothesis. Statistics. Science.
If you use A/B-tests, you are a serious marketer. Not just someone who makes things pretty, but someone who uses science to improve margins.
But, the reality in email marketing is often very, very far from this data-driven environment.
Sure, data is looked at, and a winner is established.
But thresholds and guardrails are often not built at all, or just ignored, in the quest for faster insights.
This makes A/B-testing worse than guessing in email marketing. It is about as accurate, but gives decision-makers a false sense of certainty.
So, what have A/B-tests done to me that I am so against them? And how can we use them to actually produce trustworthy results… if at all?
Beyond the checkmark
Whatever ESP (email service provider) you use probably has some sort of automated A/B-testing module. It automatically routes traffic to one or the other variation, evaluates the data, and returns a winner. Usually, the winner comes with a cute trophy icon.
This is where the whole problem starts. These green checkmarks give us the impression that everything is settled, and one variation has definitively won the test.
That is far from the truth.
I have, on occasion, tested the same exact variation twice. As in, no change at all, identical variations.
The number of times I got a “definitive” winner out of these tests is scary, and cured me once and for all of the idea that these automatic evaluation tools are reliable.
“How is that possible”, you may think?
A quick technical explanation
It’s called a “false positive”. Every A/B-test is evaluated based on statistical rules. Essentially, the bigger our sample is and the bigger the difference between both variations, the more confidence we can have in the results.
That confidence is calculated as a percentage. Now, there is no such thing as 100% confidence. So, the general rule with email A/B-tests is, if we can get a confidence score above 95%, we consider the test conclusive.
So, on its own, this means that only 5% of our “conclusive” tests are actually not conclusive. Not too bad, right?
But this is assuming that both the A and the B audience in our A/B-test are exactly identical. And they never are…
It’s never just one audience
Think about it. You create an A/B-test on an audience of 50k people. Each audience will be 25k recipients.
What would you say the chances are that these two groups of 25k are identical?
Will they have the exact same amount of high-LTV buyers? The same amount of high intent recent signups? The same amount of dead weight?
Probably not.
And, even if they did, could one send be enough?
It never is. The entire premise of concluding an A/B-test in one send is false. There are simply too many factors outside our control.
If you don’t believe me, run a few “identical” A/B-tests where both versions are the same… I promise you the number of times you will get “conclusive” results will be higher than you think.
So, how can we make these numbers trustworthy?
Rinse and repeat
The first thing to realize is that we need to run a lot more repeat tests. That is the first and most important golden rule of successful A/B-testing.
Even if one variant outperforms the other one 3:1, we always run a repeat test. Why?
You never know what else influenced the audience. If it is the thing you’re testing, a repeat test should yield the same results.
A repeat test only costs you some time, but it greatly increases the reliability of your results.
In other words, repeat tests are your #1 protection against false positives.
Remember the 95% confidence I wrote about earlier?
The below table shows by how much a few repeat tests can reduce the likelihood of false positives.
So, running no repeat tests has 20 times higher chances to produce a false positive than just running a single repeat test. Pretty crazy.
So, why is this not a bigger thing?
The need for speed
One major reason repeat tests are not a much bigger part of the conversation around A/B-tests is that this process takes more time. Because theory is cute, but reality is messy.
Sending repeat tests requires a special framework to pool them. You can’t just average the results, as sending multiple times to the same population requires even more statistical guardrails. This means a more complex formula, and a higher threshold for conclusiveness. Then, you also need to account for patterns. If you run 5 A/B-tests and A wins all 5, you’ll have a higher confidence than if A wins 2 and B wins 3. And that needs to be built into the formula, too.
TLDR: The more confidence you want, the more it will drag out your tests.
And this is something we marketers don’t like very much. We’re not a very patient bunch. We need to produce results, drive performance, and kill under-performers as quickly as possible. Time is money!
So, the prospect of slowing down the decision-making process is not something we usually enjoy. But, here is why I think that’s backwards.
Yes, we spend more time on every A/B-test. But, at the end of it, we have something tangible. A result that we have a very high degree of confidence in and can center our strategy around.
On the other hand, ending tests too quickly wastes a lot more time. We lose the time spent building the test. But, much worse, we end up feeling like we have a data-driven strategy. We have false confidence in untrustworthy numbers. And that is worse than guessing.
One strategy produces reliable results, slowly. The other one produces the illusion of quick results with no actual findings.
I know which one I prefer…
So, now, I have told you that A/B-tests don’t actually mean that much, and that you need a special statistical framework to make sense of them. So, do I just enjoy overcomplicating things, or is there actually an easy solution?
Spoiler alert, it’s the latter.
Alex’s framework for reliable A/B-tests.
One secret I have kept until now, is that I absolutely love A/B-tests. They have helped us get very useful insights into how audiences interact with the emails we send for our clients, and tighten our email creation process. A/B-tests are the best!
Now, as you can imagine, I am not sitting in front of my computer solving statistical equations all day. There is a system. And here is how it works.
Step 1: Plan
Before you judge this one as underwhelming, think about it. You’ll be testing something for the next few weeks, maybe even over a month. The potential time you can waste is huge. Make a plan for things you think will move the needle most for your audience.
Step 2: Build your evaluator
Build your statistical evaluation tool. Since AI is very good at this kind of thing, anyone can build it. Ask your favorite AI to build the proper, statistically sound formula to evaluate this correctly. That’s the most important part, and you have to really insist on it. Here is a link to an example prompt.
Fair warning, the initial UI will be rough, but it won’t take long to create and beats excel sheets. If you’d rather use a premade tool that just connects to your ESP and works from the start, you can use the one I built. I’m rolling it out very slowly, so just reach out to me and I’ll set it up for you.
Step 3: Run some A/B-tests!
This is where the fun begins. Seeing the data coming in, being able to group tests, and seeing the results in real time is an amazing feeling.
Before I end this, here is a list of DOs and DON’Ts that will save you more time.
A/B-testing DOs and DON’Ts
- ALWAYS run repeat tests
- Don’t run an inconclusive test for too long. Lack of results is a finding in itself. It just means the thing you are testing isn’t that important to your audience
- Avoid subject line tests. Open rates are a very unreliable metric. In our experience, subject line tests end up inconclusive most of the time.
- Avoid send time tests. Same reason as 3
- Focus your energy on body tests. CTA placement, showing prices or not, where to show products. These are usually the biggest needle-movers and you’ll get quick results.
A/B-tests are one of the most powerful tools we have to improve our understanding of our audience. And, once you have the right system in place, they are easy to analyze. But most E-Commerce businesses never take that time, and they end up making decisions based on a bunch of very unreliable data.
Don’t fall for the illusion of results! Building a proper A/B-testing system takes a one-off time investment, but will pay off every time you run an A/B-test. It allows you to understand your audience’s preferences, improve your conversion rate from emails, and refine your creative process.
Take it from someone who has ventured into a lot of unnecessary data analytics side quests. This one is worth it!
Data-driven email marketing is what I do and I have run hundreds of A/B-tests. If you have any questions on your setup, feel free to reach out!