How to Size an A/B Test Before You Build a Variant

I used to treat a flat a/b testing result as bad luck. Now I treat it as a design failure. Not every time, but often enough that I want the sizing conversation finished before anyone writes a line of variant code.
Here is my working hypothesis: a lot of tests that come back flat never had the statistical room to detect the lift the team was hoping for. If the smallest lift that would actually change your decision was never within reach, the test could not have told you anything useful.
So I want to walk through the sizing work step by step. We will start with why flat results are usually a sizing problem, then cover what you need before you calculate anything. From there, we will define the smallest commercially meaningful lift, turn it into a detectable effect, calculate sample size per arm with a worked example, and convert that into a run length. Then we will look at trust checks before you believe a flat result, what to do when your traffic cannot support the run, and the fast checks that catch an underpowered plan. Sizing is a design step, not an afterthought.
Flat Results Are Usually a Sizing Problem, Not Bad Luck
I used to file flat results under "the audience did not care" and move on, dodging the harder question: could that test have detected anything at all?
Going back through my own test log, my working hypothesis is that most of my flat tests were sized by convention rather than calculation, so they had no real chance of reaching significance on the lift I cared about. That is my own observation from a small sample, not a benchmark, and I will not pretend a broader figure exists.
Call it a/b testing, a/b split testing or a b split testing; the sizing math does not change, and that math is where my tests quietly failed. If you have ever agonized over t-test vs paired t-test, that confusion usually starts here.
The reframe that changed how I work: a flat result is an output of the design, so the fix sits upstream, in sample size and metric choice.
Everything below is the process I run before building a variant, working backwards from the lift I need to detect instead of forwards from my traffic.
What You Need Before You Calculate Anything
Before I open a sample-size calculator, I lock five inputs. Skip one and I am guessing with extra steps.
Baseline conversion rate on the exact primary metric, from a stable window. A launch week or peak season is a spike, not a baseline.
The smallest lift that changes my decision. A pricing test might need around 5% to ship; a checkout copy test might need closer to 15% to justify the engineering time. Those numbers are my own rules of thumb, not benchmarks.
Significance, power, and one- or two-sided testing. I default to 5%, 80%, two-sided, and I write those into the brief so someone can argue with them rather than guess.
Daily eligible traffic for the segment the test reaches. If only 20% of users can be exposed, that 20% is the number that matters.
Proof I can hold the variant stable for the run. A run length I cannot commit to is not a run length, and stopping early is underpowered with extra steps.
I settle these before I score my A/B test backlog, since a test that fails this list never belongs in the queue.
Step 1: Define the Smallest Lift That Would Change a Decision
Now I write the decision down before the test: "if this wins by X, I ship it," or "if this wins by X, I fund another quarter of this work." If no lift crosses that line, the test has no reason to run.
Then I translate commercial value into a metric lift. If 1% more conversion is worth 2% more revenue per visitor, the lift I need is the one that produces the money, not the other way around.
I used to anchor on the hoped-for lift. The gain that changes a decision is usually smaller, and that smaller number sets the sample size. Rounding matters: sizing for a 3% lift when 5% is my decision line inflates the sample for nothing, since a smaller target lift demands a much larger sample.
For a brand new funnel step with no baseline, I size for a proxy event first and pre-commit to the downstream check that proves the proxy mattered. That is how I approach add-to-cart, which is the same logic behind the sales mods that actually move revenue. I test the proxy; I ship on the downstream read.
Step 2: Turn That Lift Into a Detectable Effect
Relative or absolute? That is the next question.
A 10% relative lift on a 4% baseline means 4.4%. A 10% absolute lift means 4% becomes 14%. Same phrase, completely different experiments and sample requirements, so I now write "10% relative" or "plus 0.4 points" on every brief.
Then I check what the calculator's input actually asks for. Some want the absolute difference between rates, others a relative change. Entering 0.4 into the wrong field silently builds the wrong test, and that skipped conversion is a common way a/b split testing plans go sideways. Small baselines get punished hardest: 1% to 1.1% is a strong relative win and only 10 basis points, which needs a large sample per arm. The same discipline underpins conversion optimization data more broadly.
I keep two ideas apart: the effect I designed the test to detect, and the true average effect in the population. What I observe is not automatically the truth.
Step 3: Calculate Sample Size Per Arm (Worked Example)
Everything below is hypothetical, not a benchmark from a study. My primary metric is add-to-cart at a 4% baseline, and the smallest lift that changes my decision is a 10% relative improvement. That is plus 0.4 percentage points to 4.4%.
I write the assumptions into the brief openly: 5% significance, 80% power, two-sided, even split, one primary metric, no peeking. Change any one and the sample size moves, which is exactly why they belong where someone can argue with them.
I feed both rates and those settings into a sample-size calculator and record the per-arm figure from the calculator directly. I never shrink it because it looks big. That figure is an output of my assumptions, so the honest move is to change the assumptions or the metric, never the output.
Total sample is per arm times the number of arms, so a three-variant test needs three arms of traffic, and at a minimum the total traffic requirement triples. Multiple-comparison corrections can push it higher still. A trial length test I ran with three arms against one slow metric came back flat, and the diagnostic work that followed took weeks.
The sample-size literature also covers correlated data, which matters if arms share users or metrics move together. My tests are simple independent splits, but the formulas assume that case.
Step 4: Convert Sample Size Into Run Length
Take the per-arm figure from Step 3, multiply by the number of arms, then divide by daily eligible traffic for the segment the test actually reaches, not site-wide traffic. That is the minimum run length in days.
I add buffer for day-of-week effects and ramp-up, and I plan for at least two full weeks of exposure even when the arithmetic suggests less, often much longer. Effects can drift as novelty and primacy wear off (long-running experiments), so a three-day read proves little.
If arms share users or metrics move together, the arithmetic shifts, and a paired t test is the frame I reach for.
I sanity-check the calendar too. A test starting mid-promotion or spanning a holiday will not produce a clean read no matter how much traffic it collects.
If the arithmetic hands me a run I cannot hold, I change the design. Shortening it because I am impatient is the failure this process exists to prevent.
I write the run length and the look date into the brief before launch, not after the first disappointing reading.
Trust Checks Before You Believe a Flat Result
Distinguishing a broken test from an underpowered one is what these checks do. Before I trust any read, flat or positive, I run three checks I rely on before trusting any result.
An A/A test runs a variant identical to control. Any significant difference means my setup is broken, not that I discovered something. A/A tests should show no significant difference roughly 95% of the time when I am using 95% confidence intervals, and browser-redirect implementations are especially prone to failure here, with redirect latency and bot asymmetry as common culprits.
Sample ratio mismatch compares the split I got against the split I designed. A mismatch usually means a logging or assignment bug, and it invalidates the read.
I also check that the variance matches what I expected from the baseline window. A variance that is lower than expected can make a flat result look real, and one that is higher than expected can hide a real effect.
Our rerun of the most-cited CTA case study only counted as a wash once the setup held.
I run these before writing the conclusion, not after someone asks why a number looks odd. A flat read that fails a trust check is not a flat read at all.
If Your Traffic Cannot Support the Run, Move the Metric Closer to the Click
If the trust checks pass and a flat read still stands, I ask whether the design ever had a chance. When sample cannot arrive within a reasonable run on a purchase or renewal metric, I move the primary metric closer to the click.
Earlier funnel events carry more traffic. In my experience, add-to-cart, trial start and qualified lead fire far more often than a completed purchase, so the required sample tends to arrive much sooner. Beyond AIDA: How the Marketing Funnel Really Works in 2026 is where I check which stage each event sits in.
The trade-off is real: an earlier metric sits further from revenue, so a lift there is a hypothesis about business value, not proof of it. I verify carry-through by checking whether the downstream metric moves too, using a holdout or a read on the untouched cohort, and I never ship on the proxy alone.
I pick the earliest metric I can defend. A qualified lead usually beats a raw form view, and an add-to-cart usually beats a product page view. Before launch I write down which downstream metric must move for a proxy win to count, then check it at the end.
If even that cannot reach sample in a reasonable run, that is a genuine constraint. I change scope, change the segment or stop. I do not announce that experimentation does not work for us.
Fast Checks That Catch an Underpowered Plan
Whichever metric I land on, I run these five checks before committing to a launch date.
Eligible traffic, not total traffic. This is the mistake I make most often. If only a fifth of visitors can enter the test, sizing against site-wide sessions makes the run look five times shorter than it is.
Count the primary metrics. Three primary metrics each need their own sample. Most teams do not have three tests' worth of traffic, so I pick one and let the rest ride as secondary.
Do not stop at the first significant reading. In my experience, peeking inflates false positives, which is the statistical problem of optional stopping. That is a different failure from underpowering, but the results deck looks identical.
Freeze the variant. Changing the copy or logic mid-run throws away the sample already collected. The clean n is smaller than the logged n.
Write the brief first. Filling it in after the test ends is how a flat result turns into a story about what probably happened.
Each check takes minutes. Together they stop a plan from quietly wasting a quarter.
Sizing Is a Design Step, Not an Afterthought
The fix for all of that sits upstream. That loop, from lift to effect, effect to sample, sample to run length, then trust checks, is the design step this post walks through.
One habit ties it together: a log of the lift every test was powered to detect. Without it, a flat result is a story. With it, I can see whether the test was unlucky or badly sized, and sized is something I can fix.
Conclusion
The log is where I start the next test: keep a record of the lift every test was powered to detect, and the sizing brief writes itself.