Cold Email A/B Testing: What Should You Test First?

Open almost any cold email tool and you'll find an A/B testing feature. Open almost any outbound team's test log and you'll find a graveyard: "Subject A won, 4.1% vs 3.6% reply rate, 240 sends each." That result is noise. With 240 sends per variant and a 3% baseline, the 95% confidence interval on each arm is roughly ±2.5 percentage points. You could have flipped a coin and gotten a cleaner answer.

The problem isn't that A/B testing doesn't work in cold email. It's that most teams test the smallest variables first, at volumes that can never resolve them, while the variables that actually move reply rates by 2–3x go untested. Here's how to flip that around.

{"h2":"placeholder"}

First, the math you can't avoid

A rough, usable sample size formula for comparing two conversion rates at 80% power and 95% confidence is n ≈ 16 × p(1−p) / d², where p is your baseline rate and d is the absolute difference you want to detect. Plug in a 3% reply rate baseline and the numbers get sobering fast.

The Sample Size Reality Check

Three conclusions fall out of this table immediately.

One: if you send fewer than about 1,000 emails per week, you can only detect large differences. That's fine — large differences are the ones worth chasing. But it means you should be testing a completely different offer, not a different verb in the subject line.

Two: optimizing on positive replies requires even more volume. If your overall reply rate is 3% but only a third of those are positive, your real baseline is 1%. Detecting a doubling of positive replies (1% → 2%) needs roughly 1,600 leads per arm. Detecting a 30% lift needs about 17,000. Most teams don't have that in a quarter.

Three: stop running two subject lines against each other at 300 sends. You are not measuring copy. You are measuring which mailbox had a better week.

Test in order of effect size, not ease

Variables in cold email have wildly different ceilings. Ranked roughly by how much they can move reply rate:

• Who you're emailing (ICP and segment) — 3x to 10x swings are routine. The same email sent to VP Sales at 200-person SaaS companies versus office managers at dental practices isn't the same campaign.
• The offer and the angle — 2x to 4x. "We help you do X" versus "I noticed Y, here's what three companies like you did about it." This is the single biggest copy lever.
• Call to action — 1.5x to 2x. Asking for 15 minutes versus asking "is this worth a look?" versus asking for a referral to the right person.
• Email length and structure — 1.2x to 1.6x. Covered in detail in our post on how long a cold email should be.
• Follow-up count and spacing — 1.3x to 1.8x on total sequence replies, though this is a sequence-level test, not a per-email one.
• Subject line — 1.1x to 1.4x on replies (bigger on opens, but opens are an unreliable metric — more on that below).
• Send day and time, signature format, emoji, first-name casing — typically under 1.1x. Untestable at normal volume. Pick something sane and move on.

The practical order is: segment first, offer second, CTA third, structure fourth, everything else when you have volume to spare. Teams do this backwards because subject lines are easy to change and segments are hard.

Pick one metric before you launch, not after

Decide your primary metric in writing before the test starts. Otherwise you'll look at four metrics, find the one where your favourite variant won, and declare victory. That's not analysis, it's post-hoc rationalization.

For nearly every cold email test, the primary metric should be positive reply rate — replies that express interest or ask a question, not auto-replies, not "remove me," not "wrong person" (though referrals count as a soft positive). Total reply rate is an acceptable proxy when volume is tight, because it's roughly 3x more frequent and therefore reaches significance sooner.

Do not optimize on open rate. Open tracking is inflated by security scanners and link-prefetchers that fire pixels on delivery, and the inflation isn't evenly distributed — Microsoft-heavy lists get scanned far more aggressively than Gmail-heavy ones. We unpacked the mechanics in why cold email open rates are lying to you. A subject line test judged on opens can easily pick the losing variant.

Click rate has the same problem, worse. Treat it as a diagnostic, never a decision metric.

How to run a split that isn't contaminated

The hard part of cold email A/B testing isn't statistics — it's that your variants are being sent through infrastructure that changes underneath them. A few rules that matter more than the test design itself:

• Randomize at the lead level, not the list level. If variant A goes to leads 1–1,000 and variant B to leads 1,001–2,000, and your list was sorted by company size or scraped source, you've tested the list, not the copy.
• Run both variants across the same mailboxes, interleaved. Dedicating mailbox 1 to variant A and mailbox 2 to variant B guarantees contamination. Mailbox reputation differences dwarf copy differences.
• Run both variants in the same time window. Sequential tests — A this week, B next week — are confounded by holidays, news cycles, list freshness, and your own domain reputation drifting.
• Change exactly one thing. "New subject line and a shorter body" tells you nothing about which one worked, and the two can cancel out.
• Hold the follow-ups constant unless the follow-ups are what you're testing. Attribute replies to the variant of the first email, since that's what determined whether the thread got read at all.
• Check that both arms actually got delivered. If variant B's copy triggered a higher spam score and landed in Promotions, you measured deliverability, not persuasion. Spam-score both drafts before launch and keep an eye on bounce and placement per variant.

That last point deserves emphasis. Copy changes are deliverability changes. Adding a link, swapping plain text for light HTML, introducing words like "free" or "guaranteed" — these shift inbox placement, and placement shifts replies far more than wording does. If you see a 2x difference between two variants that read almost identically, suspect filtering before you suspect genius.

Don't peek, or peek properly

Checking results daily and stopping the moment one variant pulls ahead inflates your false positive rate badly — from 5% to something like 20–30% depending on how often you look. The simplest fix: decide the sample size up front, don't look at the comparison until you hit it, then decide.

If you can't resist, use a crude guardrail instead: only act on a result if the winning variant leads by more than 50% in relative terms and you've hit at least 500 leads per arm. You'll still be wrong sometimes, but you won't be chasing coin flips.

Also kill the habit of mining subgroups after the fact. If A lost overall but "won among healthcare companies in the 50–200 employee band," that's almost certainly noise. Subgroups are hypotheses for the next test, not conclusions from this one.

A testing calendar that fits real volume

Here's a workable cadence for a team sending roughly 2,000–3,000 emails a week across a handful of mailboxes:

• Weeks 1–3: segment test. Same email, three distinct ICP segments, ~1,000 leads each. Expect the spread to be enormous. Kill the worst segment outright.
• Weeks 4–7: offer/angle test. Two genuinely different angles on your best segment, ~2,000 leads per arm. This is your highest-value test of the quarter.
• Weeks 8–10: CTA test. Soft ask versus calendar ask, holding the winning angle constant.
• Weeks 11–12: sequence structure. Compare a 3-step to a 5-step sequence, measuring replies per lead across the whole sequence, not per email.
• Ongoing: no subject line tests until the above is stable. Then test subject lines in batches of two, on replies, with 2,000+ per arm.

One quarter, four real answers. That beats forty subject line tests that each proved nothing.

What this looks like in Sendvanta

Sendvanta splits variants at the lead level and rotates both arms across the same mailboxes, so a test isn't quietly measuring the reputation gap between mailbox 3 and mailbox 7. Engagement is classified bot-aware — verified human visits and replies are separated from security-scanner opens and clicks — so a subject line test judged on replies isn't being skewed by a Microsoft-heavy segment getting scanned twice per message. Every draft gets a SpamAssassin score before send, and deliverability health scoring across mailbox, domain, DKIM, and IP will slow or pause sending if one variant starts dragging your reputation down, which keeps a copy test from turning into an unplanned deliverability experiment.

It's honest to say that no tool fixes the sample size problem. If you're sending 400 emails a week, you don't have an A/B testing problem, you have a volume problem — test bigger swings, fewer times. The free plan ($0, 1,000 active leads, 3,000 emails/month, up to 10 of your own SMTP mailboxes) is enough to run a real segment test and find out whether your list or your copy is what's holding you back. Paid plans start at $29/mo. You can see the full feature set at /features.