Whether you're new to email marketing or have been putting it off, A/B testing is one of the most reliable ways to improve results without guessing.
Klaviyo has become one of the most widely used tools for ecommerce email marketing.
The term A/B testing is thrown about with no real explanation to companies who don’t know where to start.
Today, we want to explain A/B testing for Klaviyo to give you a better understanding of how you can use this approach in your email campaigns for short-term and long-term wins. Keep reading to learn more!
What is A/B Testing For Email Marketing?
A/B Testing for email marketing is a test variation of multiple different emails that lead to important pages on your website such as a landing page, product page, service page, etc. Essentially, you will be using opposing text, sending it out to the same audience and seeing which one performs best.
This could include making a slight change in the subject line upon multiple email campaigns and assessing the open rates, engagement rates and overall conversion rates.
One example that will sum it up well is taking arguably the most vital email of all - the welcome series. Welcoming your customers with a warm, personalised message - along with a killer CTA with the utmost urgency (and/ or an incentivised offer included) will generate masses of revenue for your company.
This doesn’t mean to say that all welcome series emails do well. Therefore, an email A/B test until you find the winning variation that is providing the highest number of sales (or results that are statistically significant) will give you the data you need to see which welcome series you should double down on moving forwards.
DON'T MISS: Know what's wrong with your Klaviyo account or how to enhance its performance? Get a FREE Klaviyo Health Audit worth $3,000
Why is A/B Testing Important For Businesses Using Email Marketing?
So, why is A/B testing important for businesses using email marketing? Well, the data you receive allows you to derive what isn’t working and what is working. Therefore, for the specific campaign moving forwards, you won’t have to guess what you think might work as you already have the performance statistics to prove that it does.
A/B testing also encourages creativity. Trying new designs, new phrases, new colours, etc in order to tweak your performance may seem like a daunting task but this is your business and you must be willing to sacrifice some time to find the sweet spot of your email marketing campaigns.
Once you know what works for your audience using the test results, you will never have to conduct too much deep research ever again. People’s purchase behaviour will change but not to a serious extent, thus, with A/B testing in each field (from sign-up forms, landing pages and many more), you will have valuable data that you can utilise forever.
What Elements of Email Marketing Do You Use A/B Testing For?
As you begin your A/B testing phase, there are various elements that you will change when testing what works for specific emails. One version of your email may work better than the other so you have to be aware of the contributing factors that could be deciding this success.
Here, we will list the different elements of email marketing you will use A/B testing for and explore them in detail to gauge the difference between each element:
Changing the From Name
The ‘from name’ is a prominent part of the email sequence when sending them off to your potential customers. It essentially shows the reader the name of the person sending the email.
If you notice that your open/ click rate is deteriorating, a change of ‘from name’ may be required. Whether you change it to a brand-related name rather than your personal name or vice versa, you will notice opposing outcomes from each process.
Email Layout
The visual aesthetic of your email needs to be pleasing to the eye of your audience, whether that be the email design or the structure of your email. Testing whether more images, more text or a bit of either work for your business is essential.
Everyone’s niche is different as the audience will differ. You can also test GIFs, testimonials and other video content. As you begin to try each variant with each other, you will begin to get a feeling of what works best as you test one variable at a time.
Subject Line
Possibly one of the most important parts of a healthy open rate is a persuasive or interesting subject line that will grab the reader's attention. Not only can you A/B test the length of your subject line (preview text), but you can change the tone of voice and CTA’s every time you send out an email.
CTA
The most important factor in converting a customer is to have a call-to-action that makes someone want to purchase your product. You can test different colours of CTAs, different phrases, different numbers of CTAs and even test the length and copy of your CTAs.
How A/B testing actually works in Klaviyo
Klaviyo's built-in A/B testing works differently for campaigns and flows, and it's worth knowing both.
Campaigns
You can test subject lines (including AI-generated variations for length, tone, or emoji use), message content (copy, images, CTA buttons, template layout), and send time. Test size is set with a slider, for example a 20%/20% split, with the remaining recipients automatically getting the winning variation once the test concludes. A 100% test size is required if you're sending based on recipient timezone. Klaviyo recommends testing two variations at a time, though you can clone a campaign to test more.
The winning variation is chosen by one of three metrics: open rate (best for subject line, preview text, or sender name tests), click rate (best for content tests), or placed order rate (only available on accounts with that metric enabled, and not compatible with personalised variations).
Statistical significance
Klaviyo won't call a winner until at least 50 recipients have received each variation. From there: 90%+ win probability is "statistically significant," 75-89% is "promising" (Klaviyo recommends retesting before acting on it), and below that it's either "not statistically significant" or "inconclusive," depending on sample size and the gap between variations. Worth knowing if you're testing on opens: Apple Mail Privacy Protection distorts open-rate data, and Klaviyo notes it may need a higher significance bar once MPP accounts for more than 45% of opens.
Flows
You can also A/B test individual emails inside a flow, not just campaigns, useful for something like your welcome or abandoned cart flow rather than a one-off send. You set a distribution weight per variation (equal, or automatic, which shifts more traffic toward whichever variation is currently winning). By default, Klaviyo picks a winner on click rate (switchable to open rate) and ends the test automatically once statistical significance is reached, or you can end it early yourself with the "Choose Winner" button.
Sources used:
- How to A/B test an email campaign
- Understanding statistical significance in Klaviyo campaigns
- How to A/B test a flow email.
Putting Together Your A/B Testing: Key Elements
Putting together your A/B test settings may seem slightly cluttered and all over the place and giving you a structured plan of how you can go about it will allow you to act more effectively when it comes to performance.
- Develop a hypothesis (choosing the variables you want to test, based on what you want to push within your business).
- Use a hefty sample size (must send your A/B testing emails to a large number of people).
- Sending times (test different send times simultaneously to see which time performs optimally).
- Hone in on sending emails that will be relevant to what you want to achieve.
- Give your emails enough time before breaking down the analytics.
- Make sure to use every tool available to you for the finest analytic breakdown.
Enjoyed this Klaviyo guide? Don't forget to join 2,000+ hungry D2C enthusiasts who lap up our weekly insider insights on eCommerce email marketing.
FAQ
How does A/B testing work in Klaviyo?
For campaigns, you test subject lines, content, or send time against a variation, set with a slider (for example a 20%/20% split), and Klaviyo sends the remaining recipients the winning version automatically. For flows, you can A/B test individual emails with a distribution weight per variation, either equal or automatic.
What can you A/B test in a Klaviyo campaign?
Subject lines (including AI-generated variations), message content (copy, images, CTA buttons, layout), and send time. The winner is chosen by open rate, click rate, or placed order rate, depending on what you're testing and what's enabled on your account.
How does Klaviyo decide when an A/B test is statistically significant?
At least 50 recipients need to have received each variation. From there, a 90%+ win probability counts as statistically significant, 75-89% is "promising" (Klaviyo recommends retesting), and anything lower is either not significant or inconclusive depending on sample size.
Can you A/B test emails inside a Klaviyo flow, not just campaigns?
Yes. You can test individual flow emails, like a welcome or abandoned cart email, with a distribution weight per variation. Klaviyo defaults to picking a winner by click rate, ends the test automatically once significance is reached, or you can choose a winner manually at any point.
Related Reads Handpicked For You
If you enjoyed this Klaviyo guide, we're pretty sure you'd enjoy the other parts in this series:
- How Does Klaviyo Work? A Guide For Your eCommerce Business
- How to create a Date-Based Property Flow in Klaviyo
- How to create a Net Promoter Score Flow in Klaviyo
- How to create a Segmentation strategy in Klaviyo
- How to create an epic Post-Purchase Flow in Klaviyo
- How to Implement Qualitative Research in Klaviyo
- How to use Klaviyo's Predictive Analytics
- How to Use Klaviyo: Learn the 5 Big Mistakes to Avoid
.svg){kind=small}
.png)
1.png)
.png)
