Last updated August 2026
A/B testing sounds simple. Show half your visitors version A, the other half version B, measure what happens, pick the winner.
There’s one problem: what if your website doesn’t get enough traffic to confidently pick a winner?
That’s not just a small-site problem. Earlier in my career, I worked on a large corporate website with far more traffic than MarketingWithDave.com gets today. Even there, only a relatively small number of pages generated enough traffic to comfortably run the kind of traditional A/B tests most experimentation advice assumes you can run.
Now think about the average small-business site, consultant, blogger, or niche publisher. If you’re getting hundreds of visits instead of hundreds of thousands, waiting for textbook statistical significance can mean running a test for months, sometimes without ever collecting enough data to detect a modest improvement.
So, should you skip A/B testing? No. But you do need to be careful about what your data actually allows you to conclude.
The Real Danger Isn’t Low Traffic. It’s False Confidence.
Suppose you test two calls to action:
| Impressions | Clicks | CTR | |
|---|---|---|---|
| Control A | 50 | 2 | 4% |
| Variation B | 50 | 4 | 8% |
Variation B doubled the click-through rate. A 100% improvement. Time to put it in the quarterly presentation?
Not so fast. That’s a difference of two clicks. If the next two visitors click A instead of B, the story flips completely. That’s the danger of percentages on small samples: the number looks enormous while the evidence behind it stays paper-thin.
Low traffic doesn’t mean your data tells you nothing. It means you have to stop asking it to tell you more than it knows.
“Just Wait for Significance” Isn’t a Satisfying Answer
The standard advice is mathematically sound: establish a baseline, decide the minimum improvement worth detecting, choose your confidence level, calculate the required sample size, and don’t decide until you’ve collected it.
Nothing wrong with that math. The problem is what happens when the calculator tells a small site it needs a sample size it will never realistically hit. Optimizely itself notes that low-conversion sites simply need more time to separate small differences from noise, and other practitioners recommend testing bigger changes or leaning on qualitative research when a conventional test isn’t practical.
Those are useful workarounds. But there’s another option worth naming directly: you can still run the experiment, as long as you’re willing to accept that “we don’t know yet” might be the correct answer. That’s a very different mindset than running a test until the dashboard finally hands you a green winner badge.
Impressions Are Your Evidence. They Aren’t the Winner.
An A/B test might split visitors roughly 50/50, but that doesn’t mean both variations end up with identical impression counts:
| Impressions | Clicks | CTR | |
|---|---|---|---|
| Control A | 1,000 | 20 | 2.0% |
| Variation B | 950 | 29 | 3.05% |
Control A got more impressions. That doesn’t make it the winner. Impressions are the opportunities each variation received; CTR (clicks ÷ impressions) is what visitors actually did with them. Here, B is performing better despite receiving fewer impressions. Whether there’s enough evidence to call B the winner is a separate question.
But impressions still matter, because they determine how much evidence you’re standing on:
- 10 impressions / 1 click = 10% CTR
- 10,000 impressions / 1,000 clicks = 10% CTR
Same rate. Wildly different confidence. Impressions supply the evidence, performance determines which variation looks better, and statistics tell you how seriously to take the gap between them.
A 50/50 Split Won’t Always Look Like 50/50
If your first seven impressions are 2 for A and 5 for B, your allocation didn’t break. Random assignment doesn’t mean taking turns (A, B, A, B); it means each visitor has roughly an equal chance of landing on either side. Small samples can look lopsided purely by chance, and totals trend toward even only as the sample grows.
Returning visitors add another wrinkle. A good experiment keeps a returning visitor on their original variation, so if people assigned to B simply come back more often, B accumulates more impressions even though the original split was fair. That’s real visitor behavior, not a broken test. If a test reaches thousands of assignments and stays dramatically skewed, that’s worth investigating, but it’s a test-integrity question, not grounds for declaring a winner.
Statistical Significance and Practical Significance Are Different Questions
Say you eventually get enough traffic to be confident that B is better. Great. Better by how much?
- A: 2.00% CTR
- B: 2.10% CTR
That’s a 5% relative lift, and with enough observations you can become very confident it’s real. But does a 5% lift move your business? Maybe. Maybe not.
Every test should answer two separate questions: how confident are we that one variation performs better, and is the improvement big enough to actually matter? The first is statistical evidence. The second is practical significance, and it’s the one low-traffic sites can’t afford to skip, because you shouldn’t burn six months proving a button change is worth 2%.
Spend Scarce Traffic on Bigger Ideas
This is where conventional low-traffic advice and my own experience line up completely. High-traffic sites can afford to test “Get Started” against “Start Now.” A small site can’t, and shouldn’t try.
Test different offers. Try substantially different calls to action. Change the positioning. Pit a product promotion against an educational resource. If B is genuinely, dramatically better than A, limited traffic has a real shot at revealing that. And if two very different approaches perform about the same, that’s a useful answer too.
Don’t Call It on Day Three
Low traffic makes early results seductive. You launch Monday; by Wednesday A has 1 click and B has 5. B is crushing it, right?
Maybe Monday and Tuesday just aren’t representative. Maybe an email blast hit the site Tuesday. Maybe you’re extracting a business strategy from six clicks. That’s why SiteSqueeze treats one full seven-day cycle as a minimum guardrail, not a statistical guarantee. Seven days doesn’t make a test valid on its own, but it makes sure the test has seen every day of the week before anyone gets to call a winner.
Not Every Test Needs a Winner
Experimentation software has trained everyone to expect a trophy: 🏆 Variation B Wins! It feels satisfying. It isn’t always what the evidence supports.
For a low-traffic site, three outcomes are all perfectly legitimate:
- Keep testing. B may be ahead, but not by enough to act on yet.
- No meaningful difference. The two approaches perform similarly within the range you actually care about, so if the new creative took three hours to build for no real gain, there’s no reason to switch.
- Inconclusive. You ran it 30 days, hit the impression budget you were willing to spend, and still don’t have enough evidence either way. That’s not a failed experiment. It’s the experiment stopping you from claiming certainty you didn’t earn.
Why SiteSqueeze Uses Bayesian Statistics
This is the problem that shaped how I built the A/B testing feature in SiteSqueeze. Instead of leaning only on a p-value, it uses a Bayesian model for CTR experiments: each variation starts with a weak, neutral prior that updates as impressions and clicks accumulate.
The math matters, but the marketer-facing payoff is simple. I’d rather tell someone “Variation B currently has a 92% probability of outperforming Control A” than hand them a p-value and hope they know what to do with it. Bayesian statistics don’t solve low traffic, though. With little data, there should still be real uncertainty. The model is built to describe that uncertainty, not paper over it.
Decide What “Better” Means Before the Test Decides for You
The other safeguard: define your minimum meaningful improvement before you get emotionally attached to whichever variation starts pulling ahead. Maybe that threshold is 10%, maybe 50%. In SiteSqueeze, I’ve set 20% as the default, adjustable per experiment. The exact number is a product decision. The principle isn’t: don’t let a statistically believable but practically irrelevant difference make your marketing decisions for you.
The Framework
- Test something worth learning, not a microscopic difference.
- Split traffic fairly, regardless of which side is leading.
- Keep returning visitors on their original variation when possible.
- Measure rates, not raw impression totals.
- Give the test a full weekly cycle before considering a winner.
- Define the size of improvement that would actually matter, up front.
- Don’t mistake an early lead for evidence.
- Use the strongest statistical evidence your traffic actually supports.
- Be willing to keep testing.
- Be willing to conclude you don’t know yet.
That last one might matter most.
Why I’m Building It This Way
SiteSqueeze didn’t start as “the world needs another A/B testing plugin.” I built it to promote my own content and offers across MarketingWithDave.com, and testing became the obvious next step: if I’m promoting something on my own site, I want to know which version actually works.
But my own traffic forced me to confront the exact problem this article is about. Pretending I had enterprise-scale data wasn’t an option, and a feature that just told me I needed hundreds of thousands of observations before it would say anything useful wouldn’t have been either. So SiteSqueeze’s A/B testing is built around a middle ground: rigorous about what the evidence supports, without pretending marketers operate in laboratory conditions.
In practice, that means 50/50 allocation, visitor assignments preserved across return visits, CTR as the measured outcome, Bayesian probability instead of a bare p-value, a seven-day minimum before any winner is suggested, and a user-defined minimum meaningful improvement. When the evidence isn’t strong enough, “keep testing” and “inconclusive” are both valid results.
It doesn’t solve low traffic. It’s designed to respect it.
You Don’t Need Enterprise Traffic to Learn Something
There’s a real difference between “I don’t have enough data to prove B is better” and “I learned nothing.” A responsible test might tell you a dramatic change looks promising and deserves more data. It might tell you two approaches are too close to justify caring about the difference. It might expose that the “winner” you were about to ship was really three extra clicks. Or it might just say there isn’t enough evidence yet, and I’d rather have software tell me that than confidently hand me the wrong answer.
A/B testing on a low-traffic site isn’t a statistical loophole. It’s making the best decision the evidence you actually have can support.
SiteSqueeze is coming soon.
I’m building a WordPress plugin designed to help marketers promote their own content, measure what works, and run practical A/B tests without pretending every website has enterprise-level traffic.
Join the list to know when SiteSqueeze launches.
