Cold-email A/B testing: build a test you can learn from

Write the decision in one sentence: “Should we replace this subject line for this audience and offer?” That gives the experiment a job. “Find our best email” is too broad: the audience, promise, body, next step and sending conditions could all change at once.

For a simple first comparison, treat the complete subject line as one variable and hold the rest of the message constant. You are testing those two lines, not proving a universal rule about word count, punctuation or curiosity. If both wording and length differ, the result cannot tell you which property mattered.

The fictional example in this guide compares a broad service label with the name of a concrete deliverable. Its hypothesis is that naming the deliverable may produce more relevant requests for it. This is a question to investigate, not a claim that one style wins.

Choose what receives an assignment: one person or one company. Keep each unit in one group for the comparison. If colleagues could compare messages or trigger company-wide stopping, assigning the company together may fit the question better. That choice also changes how the result must be counted and analysed.

For a person-level test, begin with one eligible cohort, remove duplicates and known exclusions, then assign its members randomly. NIST describes random assignment of experimental conditions to units. Do not give A last month’s list and B this month’s list and call the difference a copy experiment.

Run the groups over the same relevant period. Check whether one has received all the newer accounts, one region or a different sending schedule. Record important differences before interpretation. Equal group sizes are useful for reading a comparison, but equal totals alone do not demonstrate comparable treatment.

Preserve an assignment file containing a stable internal contact or company ID, group, assignment time and intended version. Keep personal data in the appropriate controlled system; the working brief can reference that file without duplicating the prospect list.

Define a positive human reply using examples a reviewer can apply. For this fictional offer, an explicit request for the outline qualifies. A fee question stays in a separate “needs review” category until the agreed rule resolves it. An automatic absence message, opt-out or simple rejection does not become a positive response.

Our primary measure is the share of all assigned contacts who produce at least one qualifying reply within the stated window. Count each contact once. Also report attempted sends and failures so missing exposure remains visible. Do not quietly drop difficult records after seeing which group they belong to.

Use a fixed response window relative to each contact’s planned first send, then wait until the last contact has reached its cutoff. Seven calendar days is an invented example for this guide, not a recommended duration. Record late replies separately instead of changing the cutoff to improve a result.

Instantly documents automatic replies in its default reply-rate view and an option to exclude them. Your experiment still needs its own written classification and denominator; do not assume a dashboard label uses the definition you chose.

Give each version a stable label and retain its exact subject, body and settings. Preview the fully rendered messages with representative permitted test data, including a missing personalisation field. If the two versions render differently for an unintended reason, correct the problem before assigning the live cohort.

Instantly’s A/Z documentation says assignment is random at each step and cannot be matched across steps. It also describes automatic optimisation that deactivates variants, and default distribution balancing over a campaign’s lifetime. Adding variants part-way through can therefore change who receives what.

For the simple fixed comparison here, use one tested step and keep later treatment identical, or define a complete sequence-level design whose assignment your tool can actually preserve. A person receiving A in step one and B in step two is not automatically evidence for an “A sequence”.

Confirm automatic optimisation and other distribution settings match the planned method. If they change allocation during the experiment, document that design and use an analysis suited to it. Do not label an adaptive campaign a fixed equal-split test after the fact.

Check delivery failures, unexpected queue behaviour, opt-outs and reply handling while the test runs. Statistical planning is never a reason to ignore a recipient’s request or continue an operational problem. Apply the existing contact and incident rules immediately, record the affected scope, and reassess whether the comparison remains interpretable.

Separate monitoring for harm or broken execution from choosing a winner. For this simple workflow, agree a review point and do not keep extending the run merely because a preferred version is behind. If your team wants decisions at repeated interim looks, use a statistical method designed for that monitoring plan.

Keep a dated change log: added accounts, pauses, edited copy, exclusions, routing changes and missed sends. Preserve the underlying records as well as the summary. These practical controls follow the experimental-design principle of checking measurement, avoiding unplanned changes and recording what actually happened.

Consider this invented result from 600 eligible contacts, one per company, assigned 300 to each group. All 600 have a recorded first attempt and a complete observation window. The numbers are teaching data, not an Inbox Leads campaign, a benchmark or a sufficient sample-size recommendation.

A has 15 qualifying contacts out of 300, or 5%. B has 21 out of 300, or 7%. The observed absolute difference is 2 percentage points: 7% minus 5%. The relative difference is 40%: (7% − 5%) ÷ 5%. B produced six more qualifying contacts in these records. All three descriptions refer to the same observation.

Do not write “40% better emails” in the decision log. That loses the counts, the size of the absolute difference and the uncertainty about another cohort. If A had zero qualifying contacts, the relative calculation would be undefined; report the counts and absolute difference instead.

The example also records three opt-outs in A and nine in B. Those are 1% and 3% of assigned contacts. Handle every request according to its scope, then assess this outcome alongside the primary measure. More qualifying replies does not by itself settle whether the new version should be adopted.

Before a real test, decide what difference would matter commercially and what uncertainty you can accept. Sample planning depends on assumptions about the underlying rate, the change of interest and the chosen statistical method. NIST illustrates these dependencies for proportion testing; its example formula is not a ready-made two-group outreach calculator.

If you need an inferential conclusion, use a method suited to the actual assignment unit, repeated monitoring and number of comparisons. If you cannot support that analysis, record the result as exploratory. There is no universal contact count that makes every outreach test decisive.

Use a short decision statement: what was observed, what could explain it, whether the operating checks passed, and what happens next. “Retain A; B merits another planned comparison after review of opt-outs” is more useful than a green winner badge without an explanation.

A non-decisive test can still improve the next one. You may discover that the offer needs clearer qualification, reply categories were ambiguous or the assignment record was incomplete. Fix that constraint before dividing the remaining audience into ever more variants.

Complete the brief with the person who owns the campaign and the person who will interpret it. Keep the exact versions and assignment file linked to the record. Another reviewer should be able to reconstruct which contacts were eligible, what each group received and how the outcome was counted.

Include both groups and their follow-ups in the capacity plan. An experiment changes how you allocate a justified audience; it does not create a reason to contact extra people simply to reach a target sample. When the review is complete, save the decision alongside the result so the next campaign begins with the lesson.