The State of Subscriber Email: What Our Holdout Tests Actually Prove

The State of Subscriber Email: What Our Holdout Tests Actually Prove

Everybody has an opinion on how many emails you should send to your subscribers.

But nobody backs it up with data.

So, we decided to change that - in this report, we have 43 holdout tests across 34 subscription brands and 2,195,935 recipients, measuring what happens when you change what a subscriber email can do.

I've audited hundreds of DTC brands over 18 years, and after going through this data with Zaymo, here’s our conclusion: Your results depend far less on how many emails you send, but they depend far more on what the email lets a customer do.

Key findings

Across 43 holdout tests covering 2,195,935 recipients at 34 subscription brands, Magnet Monster and Zaymo found that emails with in-email subscription actions (delay, skip, swap, one-click add-ons) reduced cancellation by 10.3% versus standard versions of the same message.

  • 17 of 43 tests showed significantly lower cancellation with the interactive treatment. Only 1 of 43 showed a significant increase.
  • Among the significant wins, the median cancellation reduction was 19%, with individual results ranging from 12.8% to 40.6%.
  • In a separate holdout test, subscribers who received billing reminders cancelled at the same rate as those who received none - and spent 10% more over their lifetime.
  • Interactive emails consistently produced lower click-through rates alongside better business outcomes, because the action completes inside the email itself.

These figures describe what happened across this test pool, not a guarantee for any single brand. The evidence, caveats and playbook are presented below.

First, what brands are actually sending

Before the tests, let’s get a snapshot of reality.

Across 22 synced brand accounts over the 90 days ending 30 July 2026:

  • 13.6% of brands averaged 1-2 email campaigns per week
  • 50% averaged 3-5 per week
  • 36.4% averaged 6 or more

So 3-5 campaigns per week is the most common operating band in subscription DTC right now.

Remember that this is a description of what brands do, not proof of what works. These are campaign objects, not emails-per-subscriber, and no causal link was found (or sought) between cadence bands and cancellation.

36% of brands launch 6+ campaigns a week while one in seven launches 1-2. After years of frequency debate, the market hasn't converged on an answer, and the test data below suggests why: the bigger differences lie in outcomes that are triggered by email, instead of send count.

And the email where that difference shows up most is one you're already sending: the billing reminder. We'll get to exactly how to rebuild it at the bottom of this report.

The benchmark: 43 holdout tests, one clear pattern

Zaymo ran interactive-versus-fallback A/B tests across their client base: one group received an email with in-email subscription actions (delay, skip, swap, one-click add-ons), the other received a standard fallback version of the same message.

Both groups got the same message at the same moment with a different capability.

The email design matters here - these tests measure the effect of what the email lets a customer do, not the impact of sending an extra email.

The publication pool is deliberately strict:

  • 43 tests across 34 brand accounts and 2,195,935 total recipients
  • Every test had at least 10,000 opens and a realised split between 45% and 55%
  • Test sizes ranged from 10,514 to 300,404 recipients (median 32,526)
  • Observation windows ran 96 to 524 days (median 207 days)

These near-even splits rule out raw exposure imbalance as the explanation.

Here's the outcome distribution on cancellation:

  • 17 tests (39.5%) showed significantly lower cancellation with the interactive treatment
  • 25 tests (58.1%) showed no statistically detectable difference
  • 1 test (2.3%) showed significantly higher cancellation

Directionally, cancellation was lower in 34 of 43 tests. When repeat tests were pooled within each brand, 27 of 34 brands still leaned lower.

Across all 43 tests, pooled cancellation was 3.031% in the interactive treatment versus 3.380% in control. That's a 10.3% relative reduction, with an approximate 95% confidence interval of 0.303 to 0.396 percentage points on the absolute difference.

Rate-adjusted across the observed windows, that's roughly 4,048 cancellations that didn't happen.

Among the 17 statistically significant wins, pooled cancellation was 3.7% versus 4.714% in control, a 21.5% relative reduction. Individual wins ranged from 12.8% to 40.6%, with a median of 19.0%.

Here’re three takeaways for you before we go further:

  1. First, the most common single outcome was "no detectable difference". That means adding useful in-email functionality rarely creates a cancellation penalty. The downside case is barely significant: one significant negative out of 43.
  2. Second, when it works, it's commercially meaningful. For a subscription brand, a 19% median reduction in cancellation among the wins is the difference between a CAC model that works and one that doesn't.
  3. Third, a caveat for that 10.3% number above: it leans heavily on the biggest tests, counts some brands more than once, and mixes very different emails and starting points. It describes what happened across this pool, not what will happen to your brand.

In practical terms: adding delay, skip, swap and add-on options to your subscriber emails is close to a free bet. It rarely hurts, and when it pays, it pays properly.

The exact changes to make are in the playbook at the bottom of this report in the section: Where interactive email earns its place, ranked by evidence.

First, why it works.

Do billing reminders cause subscriber churn?

The benchmark on its own doesn't tell you why these results happened, so let's get into the mechanism.

The single most expensive misconception in DTC subscription marketing is the belief that silence protects LTV. That if you hide the billing reminder, or strip it down to a bare "your order is coming" notice, customers will forget to cancel.

Read: Why You're Wrong About Emailing Your Subscribers: The Billing Reminder Playbook (With Brice Douglas, Zaymo)

Brice Douglas, Zaymo's CEO, has been dismantling this belief for years, and I've argued the same from the agency side: if your retention programme is designed to make customers forget you exist, your subscription business will be viewed as a scam.

Zaymo’s data now backs the principled position.

They shared a holdout test with a coffee brand where one group received billing reminders and one received none. Cancellation rates were identical between the two groups, but the reminder group spent 10% more over their lifetime.

And when the reminder leads with flexibility rather than a bare charge notice, it does more. In a test for an oral hygiene brand, their flexibility-led billing reminder produced 13% lower cancellation and a 50% increase in cross-sells versus the generic version.

Why does this work? Because of what's actually driving cancellation, i.e., too much product. We observed this across a host of brands that use Zaymo’s software.

Here’s what you need to remember - the customer still likes what you sell. But the problem is they just have three tubs of it on the shelf, because the default delivery cadence ships product faster than they can use it.

Moreover, the billing reminder lands and tells them they're about to be charged again. So if the only visible option is "manage subscription", they click it, end up in the portal, and the next step from there is cancelling .

Result? A temporary timing problem becomes permanent churn.

It also proves that you have a logistics problem and not a loyalty problem. The answer lies in making the reversible action easier than the irreversible one.

Think of it this way: 

  • A delay in product delivery solves overstock
  •  A product swap solves preference mismatch.
  • A frequency change solves pacing or timing.

When those options sit one click away inside the email itself, the customer solves their temporary problem without ever seeing a cancel button.

And the click location matters more than you'd think. Zaymo's data shows in-email actions produce roughly 20% lower attributed churn than the same options behind a redirect-to-portal link, and embedded CTAs get around 50% higher take rates than platform quick links.

As Brice puts it: "If the subscriber has to leave the email to complete the motion, you've already lost most of the lift."

Here’s a caveat on the mechanism.

Zaymo tracked delay clicks in 35 of the 43 tests, and the delay button only explains part of the retention gains. Brands where customers delayed more shipments did tend to cancel less, but the link is loose.

Which tells us something useful: giving customers control inside the email changes more than delays. It changes swaps, frequency changes, add-ons, and probably the simple expectation that this brand makes life easy rather than hard.

Two brands in the pool show how loose the link is:

  • A UK electrolyte brand cut cancellations by 19% with roughly 2 delay clicks for every cancellation avoided
  • A US men's grooming brand cut cancellations by 15.1% with nearly 39 delay clicks for every cancellation avoided

Both improved, but through very different customer behaviour. So don't turn this into "one delay saves one subscriber". It doesn't work like that.

Is click-through rate still a useful metric for subscriber email?

For years I've told brands that click rate and Revenue Per Recipient are misleading numbers to chase, because they naturally fall as your list gets bigger and older.

Interactive email affects click rate in an even more basic way. Look at these test results:

  • A home & garden brand: click-through down 20%, placed-order rate up 68%, attributed revenue up 27%
  • A health & wellness brand: click-through down 47%, cancellation down 24%
  • A fItness & nutrition brand: click-through down 39%, cancellation down 22%, one-time upsells up 37%
  • A personal care brand: click-through down 25%, cancellation down 21%, one-time add-ons up 8%

In every one of these tests, the email with the lower click rate made the brand more money.

Why? Because there was nothing to click through to. The thing the click used to lead to (the add-on, the delay, the order) happened inside the email itself.

When customers can act inside the email, click rate stops telling you how the email performed.

So if your agency still puts click rate at the top of your reports, ask them to change it. Instead, what you need to know is what customers actually did: cancellations, add-ons, orders, revenue.

The test that went the wrong way

One test out of 43 went the other way: A US-based sportswear brand saw cancellations rise from 3.3% in the control group to 4.6% with the interactive email, a 40.2% increase.

Interestingly, this test recorded no delay clicks at all. That could mean the email was built differently, the tracking failed, or the treatment genuinely backfired.

On the other side, a European skincare brand cut cancellations significantly in three separate tests (33.7%, 34.8% and 40.6%), a dietary supplement brand did it twice (18.9% and 25.9%), and a US-based nutritional brand once (18.5%, plus a second test at 16.4% that didn't reach significance).

When the same brand gets the same result three times, that's far stronger evidence than one lucky win.

A few brands also had mixed or flat results across repeat tests which tells you the other half of the story: the technology doesn't guarantee anything. How you build the email, and who receives it, decides the outcome.

Two more patterns worth knowing:

  • Brands with the worst cancellation rates to begin with tended to see the biggest improvements. That makes sense: if your billing reminders are currently pushing people to cancel, you have the most to gain.
  • Big tests and small tests won at similar rates, so the results aren't being carried by a handful of small, noisy tests.

Where interactive email earns its place, ranked by evidence

"Interactive email" isn't one tactic. It's a capability you apply across the lifecycle, and the evidence is stronger in some moments than others. Here's our honest ranking.

1. Billing reminders and upcoming-order notices. The strongest evidence, the clearest reason why it works, and the 43-test benchmark above. Rebuilding this email should be your top priority. 

2. Abandoned cart and post-purchase. A home & garden brand's abandoned-cart test drove a 68% higher placed-order rate and 27% more attributed revenue. A specialty retail brand's randomised post-purchase test lifted placed orders 25%. Worth noting what doesn't need interactivity: Brice is upfront that simple cart reminders are usually enough when the customer has already built a cart. The win comes from making the purchase itself easier, not from dressing up the reminder.

3. Cross-sells inside subscriber flows. Published ROI figures across Zaymo's named case studies run from 5X to 16X: Gratsi at 16X, Just Ingredients at 15X, AKT London at 13X, Cornbread Hemp at 12X, Rasa at 11X, Create Wellness at 10X. Each brand calculates ROI its own way, so don't average them, but the pattern is consistent: customers complete the action when it's one click away. Fatty15 also reported 67.4% higher BFCM revenue and 69.6% more orders year on year with interactive email in the mix; that's a year-on-year comparison rather than a controlled test, so treat it as directional.

4. Reviews. 1906, a supplements brand, collected 42% more reviews in a controlled test by letting customers leave the rating and review inside the email. As Brice puts it, customers are willing to leave reviews, but a link somewhere means most don't take the time.

5. Pre-renewal upgrades. A YOCTO-led collagen brand generated more than £700,000 in annual upgrade revenue from emails sent before the first and second renewal. That figure comes from tracked revenue rather than a controlled test, but it's a big number against a moment most brands leave completely silent.

6. Welcome and winback. Midnight Hour's welcome quiz was completed by 55.8% of openers. More striking: their winback quiz, sent to their coldest, most disengaged segment, was completed by 36.8% of openers and 12.5% of everyone who received it. Customers who ignore standard campaigns will still complete a quick, relevant interaction. These numbers measure engagement and customer data collection rather than revenue, but one in eight of your "dead" segment completing an in-email experience should change how you value that segment.

7. Post-cancellation. One brand wins back roughly 0.5% of its cancelled subscribers every week with one-click reactivation emails, no discount attached. A small percentage, but it adds up week after week, on a segment most brands write off entirely.

The playbook: what to actually do with this

Here's what I'd implement with this data, in order.

1. Rebuild your upcoming-order email around flexibility.

  • Put delay, skip, swap and frequency-change options directly in the email, not behind a portal login
  • Make upcoming rewards visible (gift on 2nd order, milestone perks)
  • Reiterate the benefits of staying subscribed

Read: Why Your eCommerce Subscription Revenue Has Plateaued

The reversible action must be easier than the irreversible one. This is the highest-leverage email in your entire programme.

2. Stop optimising subscriber emails for clicks. Report completed actions, cancellation rate, add-on take rate and placed orders. If CTR drops while cancellations drop, the programme is working.

3. Add one-click add-ons to billing reminders. Your subscribers already have saved payment details, so the usual six-step checkout can become a single tap. This is where the 8-37% upsell lifts in the case studies came from.

4. Time your upsells by order count, not the calendar. Around 70% of subscription churn happens between orders 1 and 3. Before order 3, your emails should remove friction and build the habit. After order 3, the customer is settled and the bigger risk is not selling to them enough. Upsell on behaviour, not time.

Read: How to Email Active Subscribers to Grow LTV for DTC Subscription Brands

5. Put an interactive quiz in front of your dormant segment. Before you pay Meta to re-acquire customers you already have, test whether a quick in-email experience wakes them up. Midnight Hour's 12.5% completion rate from cold recipients is the reference point.

6. Run real holdout tests. This entire report exists because Zaymo tested with proper controls. If you implement any of this, hold back a control group with a near 50/50 split, decide upfront how long you'll measure for, and make sure the audience is big enough to detect the improvement you care about.

What we still can't claim (and won't pretend to)

This report doesn't answer:

  • Whether one extra weekly email costs you unsubscribes or spam complaints
  • The right number of sends per week
  • How much extra revenue each additional weekly email actually brings in
  • Whether the revenue from sending more is genuinely new, or just pulled forward from next month
  • Whether sending more improves lifetime value

Answering those properly would mean randomising how many emails each subscriber receives and tracking the outcomes over an agreed window, across many brands. That's the dataset we'd like to build next.

But, here's what we can say.

Across 43 large controlled tests where both groups received the same email, giving customers useful actions inside the email rarely made cancellation worse, and often made it meaningfully better, with proven wins ranging from 12.8% to 40.6%.

Bottom line: Subscriber email is moving from sending messages to completing useful actions inside the inbox.

Research Methodology

43 interactive-versus-fallback A/B tests with cancellation outcomes, drawn from 34 brand accounts and 2,195,935 recipients. Every retained test had 10,000+ opens and a realised treatment share between 45% and 55%. Observation windows were cumulative, ranging from 96 to 524 days (median 207). Both groups received the email; results measure email experience, not send frequency. Statistical significance assessed at a two-sided 95% threshold; "no detectable difference" does not mean "no effect". Significance counts move as outcomes accrue (the 4 August refresh moved one test from non-significant to significant); directional counts are steadier, and all figures here use the 4 August 2026 extraction. Published case-study figures are reported using each brand's original definitions and are kept separate from the audited aggregate.

Frequently asked questions

Do billing reminders cause subscriber churn?

No. In a controlled holdout test with a coffee brand, subscribers who received billing reminders cancelled at exactly the same rate as those who received none - and spent 10% more over their lifetime. When the reminder leads with flexibility rather than a bare charge notice, it performs even better: an oral hygiene brand's flexibility-led version produced 13% lower cancellation and 50% more cross-sells than the generic email. The risk isn't the reminder. It's a reminder whose only visible option is "manage subscription".

Does interactive email reduce subscription cancellations?

Usually, and it almost never increases them. Across 43 holdout tests, 17 showed significantly lower cancellation with in-email actions, 25 showed no detectable difference, and only 1 showed an increase. Pooled across all tests, cancellation was 10.3% lower in the interactive treatment, and among the significant wins the median reduction was 19%.

What should replace click-through rate in subscriber email reporting?

Completed actions, cancellation rate, add-on take rate and placed orders. When customers can delay, swap or add products inside the email itself, there is nothing to click through to - in these tests, click-through fell by 20-47% while revenue, orders and retention improved. If CTR drops while cancellations drop, the programme is working.

How many emails should subscription brands send each week?

Across the 22 brand accounts we snapshotted, 50% send 3-5 campaigns per week, 36% send 6 or more, and 14% send 1-2 - and no causal link was found, or sought, between cadence and cancellation. Answering the frequency question properly requires randomising send volume per subscriber across many brands. What the 43 tests do show is that what the email lets a customer do matters more than how many you send.

What counts as an interactive subscriber email?

An email where the subscription action completes inside the inbox: delaying or skipping a shipment, swapping a product, changing frequency, or adding a product to the next order in one click. The distinction matters commercially. In-email actions produced roughly 20% lower attributed churn than the same options behind a portal redirect, and embedded CTAs saw around 50% higher take rates than platform quick links.

Related reads:

SHARE
  • Twitter
  • Facebook
  • Linkedin
  • Email

Keep on Reading

State of the DTC Retention Agency: 2025 Report

by
Adam Kitchen

How to Set Up Klaviyo's Expected Date of Next Order (EDNO) Flow

by
Andrew Langhorn

Email Marketing Deliverability Bible for eCommerce Stores | UPDATED 2024

by
Adam Kitchen with Nick Koreck
Monster head
BOOK A DISCOVERY CALL