I've concluded that most A/B testing over email marketing is a waste of time.
Yes, hot take, but hear me out.
This is based on working with close to 200 DTC brands over email, and auditing in the region of nearly 500.
Most A/B testing is generally implemented on the front end of the website, to convert first-time visitors into buyers.
Experimentation holds credence there. It can plug the gap in a leaky bucket, leading to tens of thousands in incremental revenue.
With email, that's not the case, and there's a very specific reason why.
Your customers already know who you are.
You don't need to run endless experiments to "convince" them to buy from you again.
You just need to stay top-of-mind with engaging content.
After working with close to 200 brands and auditing in the region of nearly 500, guess how many times I've come across a brand with a consistent, effective A/B testing protocol over email that drives incremental revenue.
Zero.
I'm not kidding. There have been some decent efforts, but the juice is seldom worth the squeeze.
Why email A/B testing fails so consistently
There's a fair few reasons for this.
- Subscribers already know who you are. Running tests to convince them to buy again is often wasted effort. You'd get more from spending that time being creative in your marketing instead.
- Most DTC brands lack the repurchase rate to reach statistical significance. Anything above a 30% repurchase rate in 12 months is considered exceptional in this industry, and most brands sit well below that.
- Sample sizes are usually too small to derive meaningful learnings from, for most brands.
Those are the core reasons A/B testing fails consistently over email, campaign by campaign.
I'd put holdout tests over campaigns, and send-time optimisation, in the same bucket.
Both lead to transient findings that seldom hold over the long term, because they're influenced by constantly shifting variables that distort their credibility. A winning send-time one quarter is rarely still winning the next, once your list composition, your content and the season have all moved on.
Bold statements, I know, but it's backed by enough evidence across close to 500 audits to make the claim.
I wrote something similar a few years ago, when I said most brands I'd worked with had no real testing protocol over email at all, and that the only things worth ruthlessly testing were signup forms, abandoned carts and post-purchase upsells, not subject lines and campaign copy. That argument still holds. This is the sharper, narrower version of it: of those, your pop-up is the one with the best risk-to-reward ratio, because it's the only lever doing two jobs at once.
Am I against all testing?
Absolutely not.
I believe in testing one thing, relentlessly: pop-ups.
There are only two ways you can influence a brand's results over email:
Drive people back to the website to buy again
Get more people on the list to buy again
Pop-ups are unique in that they don't just drive the second one. They also directly influence converting first-time visitors into buyers, which is where the majority of your testing effort should be focused in the first place.
Pop-ups tend to fall under the email marketer's jurisdiction as their responsibility, and they're a key lever you should be pulling to drive conversion and build the brand's database.
There are a huge number of variables you can test across multiple parts of the customer journey, which makes pop-up testing a non-stop source of incremental gains, in a way campaign testing simply isn't.
What to actually test on your Klaviyo pop-up
A few ideas worth testing monthly:
Offer (always number 1)
Zero-party data capture
Micro-commitments
Timing
Page scroll
Test your pop-up relentlessly, with the focus on driving conversion and increasing list growth, and keep a diligent log of your findings.
If you want the fuller menu of pop-up formats and targeting setups to test these variables against, exit-intent, cart pop-ups, zero-party data capture for unengaged subscribers and so on, I've catalogued 11 advanced pop-up strategies for ecommerce growth separately.
In Klaviyo specifically, the signup form builder is where this testing actually happens, worth bookmarking alongside Klaviyo's own blog and our guide to how Klaviyo works if you're still finding your way around the platform.
It's the main focus area that will genuinely move the needle for you as an email marketer, in a way chasing campaign-level significance never will.
FAQ
Should DTC brands A/B test their email campaigns?
In most cases, no. Most brands don't have the repurchase rate or sample size to reach statistical significance on campaign-level tests, and subscribers already know the brand, so there's little left to "convince" them of through a subject line or CTA test.
What should you test instead of email campaigns?
Your pop-up. It's the one lever that does two jobs at once: converting first-time website visitors into buyers, and growing the list of people you can email again.
What pop-up elements are worth testing?
Offer, zero-party data capture, micro-commitments, timing and page scroll depth are the variables worth testing monthly, logged consistently so you can see what's actually moving conversion and list growth over time.
Are holdout tests and send-time optimisation worth running on email campaigns?
Generally not worth the effort. Both tend to produce findings that don't hold over the long term, because list composition, content and seasonality keep shifting the variables underneath the result.
Most of what gets called "email testing" is effort spent trying to convince people who already know you.
Your pop-up is the one place that effort actually compounds, because it's working on people who don't know you yet.
Test it like your revenue depends on it. For most brands, it genuinely does.
P.S. if your agency's monthly report is full of subject-line test results and light on anything about your signup form, ask them why.
Want more of this in your inbox? Subscribe to our newsletter and YouTube channel for weekly DTC retention strategies, extracted straight from the brands we work with. No fluff, just actionable plays.
Related reads
.svg){kind=small}
1.png)
.png)
