If you want to test cold email subject lines with only 100 or 200 contacts, the trick isn’t a fancier tool. It’s knowing what a small batch can and can’t tell you.
Some advice on this topic comes from software companies built for people sending thousands of emails a week. Their rules may assume huge lists, where a few extra opens barely matter. If you’re a freelancer pitching clients, a job seeker emailing hiring managers, or a small business owner reaching out to local companies, your list looks nothing like that. The good news is that a small list can still teach you something real. You just need a cleaner method, and AI can handle the tedious parts for you.
Contents
Why Small Lists Fool Us So Easily
Picture two subject lines sent to 100 people each. Version A gets 5 replies. Version B gets 9. It feels like B won by almost double.
But replies at this scale are rare events, and rare events bounce around a lot. Send the exact same subject line to two different groups of 100 and you could easily see 5 replies in one and 9 in the other, with no real difference at all. That’s not a flaw in your email. It’s just how small numbers behave.
Three traps to watch for:
- Counting opens instead of replies. Some email apps pre-load images automatically, so some “opens” are software, not people. Vendor guides on this, such as Instantly’s explainer on open tracking accuracy, discuss why open metrics may be wrong. In my view, replies are a more direct signal.
- Trusting one round. One batch is one roll of the dice.
- Changing several things at once. If the subject line, the first sentence, and the send day all changed, you can’t know which one mattered.
Small-List Testing vs. Big-List Testing
| Big sender (thousands per week) | Small sender (100–300 contacts) | |
|---|---|---|
| Main signal | Open rate and reply rate | Reply rate only |
| Variants at once | 5–10 is fine | 2, at most 3 |
| One round is enough? | Often yes | Rarely |
| Biggest risk | Spam filtering | Mistaking luck for a winner |
| Best tool | Built-in A/B software | A simple spreadsheet |
| What a “win” looks like | A small lift | A big, repeated gap |
The right-hand column is the one you’re living in, and its variant counts are my own suggestion rather than a published standard. Everything below is built for it.
Setting Up a Fair Test
The Two-Version Rule
The simplest way to test cold email subject lines on a small list is to pick exactly two and change one thing between them. Not three versions. Not five. It’s tempting to try everything at once because each send feels expensive, but a crowded test gives you a blur instead of an answer.
With 200 contacts, two versions means 100 people per version. Add a third and you’re down to roughly 67 each, which is, in my view, too thin to say much.
Good single-variable pairs look like this:
- A plain question vs. a plain statement
- A subject line that names the recipient’s company vs. one that doesn’t
- Short (about 3 words) vs. medium (about 6 words)
Everything else stays identical: the email body, the sender name, the signature, and the day and time.
Split the List Fairly
If you send Version A to the first 100 names and Version B to the last 100, you may be comparing two different kinds of people without realizing it. An alphabetical list, for example, can bunch up similar companies.
A fair split is quick to do in a spreadsheet:
- Put all your contacts in one column.
- In the next column, type
=RAND()for every row. - Sort the whole sheet by that random column.
- Label the top half A and the bottom half B.
Send both versions in the same session, ideally within the same hour. That way timing can’t explain any difference.
Build the Reply Tracker
Make a sheet with these columns: Name, Company, Version (A or B), Date Sent, Replied (Y/N), Reply Type, Notes.
For Reply Type, use only four labels:
- Interested: asks a question, wants a call, wants details
- Not now: polite no, or “contact me later”
- No: a clear decline or unsubscribe
- Automatic: out-of-office or bounce (don’t count these as replies)
That last label matters. Auto-replies inflate your numbers just like machine opens do.
If you get a lot of messy replies, let AI sort them. Paste in the replies, with names removed, and use this:
Below are replies to a cold email. For each one, give me one label: Interested, Not now, No, or Automatic. Then give a one-line reason.
Rules:
- Use only these four labels.
- If a reply is unclear, label it "Unclear" and tell me why instead of guessing.
- Do not rewrite or summarize the reply beyond the one-line reason.
Replies:
[paste replies here]I wrote this prompt myself as a suggestion; no source publishes it as a standard.
Check a few of the labels yourself. AI can do this kind of sorting well, but it can misread sarcasm or short answers like “sure.”
What Counts as a Real Difference
Here’s a plain way to judge your result. The numbers below show how big a gap you’d need with 100 emails per version before it stops looking like luck. They come from Fisher’s exact test, a standard method for comparing two small groups.
| Version A replies | Version B replies | Chance it’s just luck* |
|---|---|---|
| 5 | 10 | about 28% |
| 8 | 16 | about 13% |
| 10 | 20 | about 7% |
| 5 | 15 | about 3% |
*Each row assumes 100 emails per version. Smaller percentages mean the gap is harder to explain by chance.
The pattern is the lesson. Doubling your replies can still be well within luck, as in 5 vs. 10. In this table, only the gap closest to triple (5 vs. 15) falls near 3%, so you may need such a gap, or you need to repeat the test and see the same winner again.
You can check your own numbers for free with GraphPad’s quick calculator. Enter replies and non-replies for each version and pick Fisher’s exact test.
A practical decision rule for small senders:
- Gap is tiny or the lines are close: call it a tie and keep the shorter, plainer one.
- Gap is large, but only one round: treat it as a lead, not a win. Run it again.
- Same version wins in two separate rounds: you’ve earned the right to make it your default.
Using AI, and Checking Deliverability First
Use AI for Candidates, Not for Verdicts
In my view, AI is useful at the start of a test, when you need good options, and weak at the end, when you need to decide who won. Let it create the two versions, then let your tracker pick the winner.
Write 6 subject lines for a cold email to [role] at [type of company]. I'm [who you are, one line] reaching out about [one-sentence reason].
Rules:
- Under 7 words each.
- Plain language, no hype, no emojis, no all-caps.
- Do not invent facts about the recipient or claim results I haven't told you.
- Group them in pairs where only ONE thing differs (for example, question vs. statement), so I can test them fairly.
The pairing rule is the key. It hands you ready-made A/B tests instead of six unrelated guesses.
Personalize Without Making Things Up
If you want a subject line that refers to something real about the recipient, AI can help you find it, as long as you give it real material. Paste the company’s about page or a recent announcement and ask:
From the text below, list up to 3 specific facts about this company that I could mention in one short line. Quote the exact words you're relying on. If the text doesn't contain anything useful, say so.
Text:
[paste here]Asking for the exact quote keeps the AI honest. If it can’t point to the words, don’t use the fact.
Check Deliverability First
If almost nobody replies to either version, the subject line may not be the problem. The email might be landing in spam before anyone sees it.
Before you test, run a quick check:
- Send your email to two personal accounts (one Gmail, one other) and see where it lands.
- Make sure your sending domain has basic email authentication set up. Look up your email provider’s setup guide for SPF, DKIM, and DMARC.
- If you send from a brand-new domain, in my view it’s safer to start with a few emails a day instead of 200 in a single burst.
Fix these first. A great subject line can’t help an email nobody sees.
The Subject-Line Rule You Can’t Test Around
Whatever wins your test, the FTC: CAN-SPAM Act Compliance Guide for Business requires that a subject line not be deceptive and accurately reflect the message content. A “Re:” or “Quick question about your order” subject on a first email that is really a pitch is the kind of line to leave out of your variants. The same guide requires a valid physical address and an opt-out you honor within 10 business days.
A Three-Week Routine You Can Actually Keep
Here’s a schedule that fits a small list (my suggestion, not a standard):
Week 1: Send Round 1 (A vs. B) to a random half of your list each. Log replies for five to seven days before judging. Late replies count.
Week 2: Take the likely winner and test it against a fresh challenger on a new group of contacts. Change one thing again.
Week 3: Look at both rounds side by side. Keep the line that held up, write it in a “what works” note, and start the next question: maybe length, maybe the first sentence of the body.
Over a couple of months, you may have a short list of patterns that came from your own audience, not from a vendor’s average. Keep that “what works” note somewhere you’ll actually see it, such as the first tab of your tracking sheet.
Each entry only needs three things: the two versions you compared, how many replies each got, and what you decided to do next. When you come back to it in a few months, you won’t have to repeat a test you already ran, and you’ll be able to see which questions are still open.
Questions About Cold Email Subject Line Tests
Can I really test cold email subject lines with only 100 emails per version? Yes, but mostly when the gap is big or the same result repeats. A single small gap proves very little.
Should I ignore open rate completely? Not entirely. It can hint at whether your subject lines are being seen at all, but don’t pick a winner from it alone.
How long should I wait for replies? In my view, at least five to seven days. Some replies may come after a few days, so stopping early can favor whichever version happened to get a quick response.
What if both versions get zero replies? Check deliverability first, then look at the email itself: the offer, the length, and whether you’re asking for something small.
Is it okay to use AI to write these emails? You can use AI to draft and organize. Just make sure every claim in the email is true and you’ve read the final version yourself.
Start With One Clean Test
Pick one pair, split your list randomly, send both in the same hour, and log replies in a simple sheet. That’s enough to begin. When you test cold email subject lines this way, a small list stops being a handicap and becomes a quick, low-cost way to learn what your own audience responds to.
Leave a Reply