What Is A/B Testing in Cold Email?
Also known as Split testing, A/B test, Email A/B testing
A/B testing in cold email is the practice of sending two versions of an email that differ in one element, such as the subject line, opener or call to action, to randomly split groups of similar prospects, then comparing results. In cold outreach the winner should be judged on positive replies or meetings, not opens, because open tracking is unreliable.
Put this into practice with our free tool. No sign-up needed.
Open the toolA/B Testing explained
Most cold email tools let you add variants to any step and split contacts between them automatically. The key is changing one thing at a time. If version B has a new subject, opener and CTA, you won't know which change caused the difference. Test the elements with the biggest likely impact first: the offer, the targeting angle and the CTA usually matter more than small wording changes.
Sample size is the hard part. Cold email reply rates are low, so a few replies of difference between variants can be pure chance. A handful of extra replies on a few hundred sends proves little. Run tests long enough to collect a meaningful number of positive replies per variant, and be suspicious of small differences.
Because open rates are inflated, subject line tests judged on opens are misleading. Use reply rate or positive reply rate even for subject line tests. Keep a simple log of each test: the hypothesis, the variants, sample sizes and the result. Over time that log becomes your playbook for a segment.
A/B Testing example
A team tests two first emails on 1,600 IT managers, split evenly. Version A leads with a cost-saving angle. Version B leads with reducing on-call alerts. Everything else is identical. After the full sequence, A gets 9 positive replies and B gets 22. That gap is large enough to act on, so B becomes the control and the team next tests CTAs.
How to run an A/B test in cold email
Write a hypothesis
State what you are changing and why you expect it to win, such as a pain-based opener beating a compliment opener.
Change one element
Keep everything identical except the element under test, including list segment, sending schedule and mailboxes.
Split randomly
Let the tool assign contacts randomly so both variants reach similar prospects at similar times.
Wait for enough data
Run the test until each variant has a meaningful number of positive replies, not just a few hundred sends.
Record and roll forward
Log the result, make the winner the new control and test the next element against it.
Common mistakes
- Declaring a winner after a few replies of difference, which is often random noise in low-reply campaigns.
- Changing several elements at once, so you can't tell which change produced the result.
- Picking subject line winners by open rate, which privacy features and scanners inflate.
- Testing tiny wording tweaks before testing the offer, angle or targeting, which matter far more.
A/B Testing FAQ
Start with the elements that change the most: the targeting angle or pain point, the offer and the call to action. Subject lines and small wording changes usually have less effect. Big changes produce clearer differences, which helps with small sample sizes.
Enough that each variant collects a meaningful number of positive replies, not just sends. With low reply rates, that often means several hundred to a few thousand prospects per variant. Small gaps on small samples should be treated as inconclusive.
Not reliably anymore. Apple Mail Privacy Protection and corporate security scanners trigger opens automatically, which distorts the comparison. Judge subject line tests on reply rate or positive reply rate instead.
A/B testing compares two versions that differ in one element. Multivariate testing changes several elements at once and measures combinations, which needs much more volume. Most cold email programs lack the volume for multivariate testing, so A/B testing is the practical choice.