← All posts

How to QA an A/B Test Before You Trust Its Results

Professional header image for step-by-step guide: How to QA an A/B Test Before You Trust Its Results

I used to treat A/B test results like a headline: open the dashboard, spot the lift, move on. Then I shipped a win that was not real, and spent months watching teammates quote it in meetings. An untrustworthy result is worse than no result, because it gets shipped and then treated as evidence for a year. That is why I now use a fixed pre-read ritual for a b testing. Before I read a single outcome metric, I check the boring plumbing that makes the number trustworthy.

In this post, I will cover why an untrustworthy result is worse than no result, what a silent test failure looks like, and four checks to run in order: verify the traffic split, confirm tracking fires at the rates you assumed, look for overlapping tests sharing the same traffic, and lock metric definitions while watching for mid-flight drift. I will also walk through the MSN carousel case, explain what to do when one of the checks fails, and sum up my pre-read ritual. If you run experiments, this is the sequence I wish I had followed sooner.

Why an Untrustworthy Result Is Worse Than No Result at All

A wrong result does not stay in the experiment tool. I have watched a broken test number move from a dashboard into a planning doc, then into a roadmap debate, then into a quarterly review, still getting quoted long after the test was archived. Nobody remembers it was flawed. They remember the number.

No result costs me a week. A false result costs me a quarter of shipping in the wrong direction, and it quietly trains the whole team to trust a broken number. I lived a version of this when we tested the internet's most famous CTA case study on our own traffic and got a wash.

That is what I mean by a silent test failure. The test runs, the dashboard fills, the confidence interval clears, and nothing pages anyone. There is no alert for a lie that looks like a finding.

The authors of the KDD 2019 paper on sample ratio mismatch ran experiments across four software companies and more than 25 products used by hundreds of millions of people, and they still hit this class of bug. Complexity does not protect you. Ritual does.

So the fix is not a stronger statistical test or a nicer dashboard. It is a fixed sequence of checks I run before I let myself read the outcome at all.

What a Silent Test Failure Actually Looks Like

I call a result a silent test failure when it clears the usual validity test. The math is fine. The data underneath is not, because of bad assignment, broken instrumentation, or a definition that changed mid flight.

The clearest version is a sample ratio mismatch, or SRM. The KDD 2019 authors define it plainly: the observed sample ratio differs from the expected one. Say I plan a 50/50 A/B split testing run and a large sample lands at 47/53. Something happened that I do not understand. That example is mine, not theirs.

Their best metaphor is fever. A fever says something is wrong without saying what. Detection is a simple statistical check, the kind of mechanics covered in T-Test vs Paired T-Test: A Growth Practitioner's Guide. Finding the root cause is slow and hard.

I cannot shrug it off. Ignore an SRM without knowing its cause and a bad change can look good and ship, or a good one can look bad and die. Both are expensive.

One honest note: verifying everything before reading the outcome is my framework, not a finding from that paper. I have just watched it save enough quarters that I stopped reading results first, which is why the first thing I check is the split.

Check 1: Verify the Traffic Split Before You Read a Single Metric

The first place I look is the split itself. Every A/B test rests on one assumption above all others: users were assigned to buckets randomly and independently. Break that and every metric below it is meaningless, no matter how clean the chart looks.

I compare expected versus observed counts per variant with a chi-square style check. On a 50/50 split, the ratio should sit near 50/50, with wobble that shrinks as the sample grows. I run it from day one, not at read time, because a split that breaks at launch gets worse, not better.

Failures look like a variant count drifting from the planned ratio, a control bucket losing users on mobile only, or counts that never add up to the exposed population I expected. Dashboards hide all of it: bot and crawler traffic landing unevenly, a redirect dropping part of one variant, a tracking script failing for one bucket, users logged as exposed without ever seeing the treatment.

My rule is blunt: fix the split or kill the test. I never write "noted a small SRM, moving on," because that sentence is how a broken experiment becomes a shipped decision. It is also the first filter in how I score my A/B test backlog.

Check 2: Confirm Tracking Fires at the Rates You Assumed

Before I read a metric, I check that the events behind it actually fired. When I scope an experiment, I quietly assume roughly X percent of exposed users will fire the primary event, that the funnel step above it converts at an expected rate, and that revenue events land in the warehouse within a known window. Nobody writes those assumptions down, and that is the problem.

So before I read results, I build a small expected versus observed table for each key event, split by variant. Not for significance. I want a gap wide enough to prove the event is not firing the way I assumed.

A gap between variants is a bug until proven otherwise, not a finding. If the primary event fires for 30 percent of control and 18 percent of variant, my first hypothesis is a broken code path, not a 12 point engagement drop.

Causes I have hit: ad blockers and client side script failures, consent banners that block one path, older app versions lagging behind a new tracking SDK, and events that drop on only one variant because of a rendering condition. I still run the table on tests I feel sure about, including the shorter onboarding test I ran twice.

To be clear, this is my own practice, not something the KDD paper covers. Treat it as a starting point, not a published standard.

Check 3: Look for Overlapping Tests Sharing the Same Traffic

That check catches instrumentation that broke. This one catches something subtler: a clean result that is not mine to claim.

The cleanest looking result I ever killed came from a test running on the same page as another test I had forgotten about. Users in both got a bundled treatment, and I could not attribute the change to either one.

Before reading, I check the platform for mutual exclusion conflicts and for tests hitting the same page, funnel step, or population at the same time. Anything touching the same surface in the same window is a suspect.

Overlap does not have to conflict to hurt you. Two tests splitting the same pool of eligible users leave each one underpowered, so both land inconclusive while the real effect hides in the noise.

Then there is the team problem. I have seen three tests launched the same morning by three people who never compared notes. A shared calendar or a short standup on what is live costs nothing and prevents it.

My rule is one meaningful change per user per surface at a time. If I cannot separate two tests, I do not read either result as causal, no matter how tempting the chart is. That same discipline sits behind how to find growth where everyone else misses it.

Check 4: Lock Metric Definitions and Watch for Mid Flight Drift

If the overlap sweep comes back clean, I do one more thing before reading anything. Metric drift is the sneakiest failure on this list because it produces a chart that looks exactly like a real effect. Revenue gets redefined on day four, a logging event gets renamed, a filter changes, and the average moves for reasons that have nothing to do with the variant.

I freeze the primary metric, guardrails, filters, and time window before launch and write them in the test doc. If a definition changes mid flight, it gets logged with a timestamp. I also watch for population drift: a new geography, device mix, or traffic source arriving mid test can reshape both buckets differently, and that is a mix shift dressed as a treatment effect.

Engineering deploys are the other culprit. A release touching logging, load order, or the event pipeline can change what gets recorded without anyone calling it an experiment change.

If a definition or pipeline did change, I split the data at the change point and treat the earlier period as a separate test. Two honest halves beat one dishonest whole. That is the same discipline behind our seven-way checkout A/B test, where the biggest lessons turned out to be about testing itself.

The MSN carousel case from the KDD 2019 paper is the one I keep coming back to, and it picks up right where the definition checks leave off. A team increased the number of carousel cards from 12 to 16 and engagement dropped significantly against their expectation. The paper notes the experiment had enough statistical power, correctly logged telemetry, and a trustworthy platform, then asks one short question: what happened?

I use it for a single lesson. A significant result sitting on healthy looking infrastructure is not proof the change caused the effect. It is a prompt to diagnose. That is why I check the split, the fire rates, the overlap, and the definitions before I interpret a direction. The result alone cannot tell me whether I am looking at a product effect or an instrumentation artifact. I got the same lesson from our pricing test where showing all three tiers lost to the single card, a counterintuitive outcome that only made sense after I checked the setup.

I am not going to claim the paper's resolution here. If you want the diagnosis and the taxonomy the authors built, read the paper itself, because the value is in how they walked through the detection and root cause work.

My takeaway is simpler: a surprising result is a signal to interrogate the pipeline, not a signal to ship.

What to Do When One of the Checks Fails

When a check fails, I stop reading the outcome. I do not report or act on it while the problem is unresolved; a deadline does not make a broken number less dangerous.

Then I triage by cause. A known cause with a fix gets fixed and rerun, sometimes from scratch. An unexplained SRM gets thrown out entirely, even if the sample is large and the result is convenient. The KDD authors warn that ignoring an SRM without knowing the root cause lets a bad change ship as a win.

There is a middle path for tests that ran for weeks before the problem surfaced. I fix the issue and restart the timer, or extend the run and use only the clean period. I say plainly that the sample is smaller and the power is lower, the same honesty as our post-mortem on a null result.

Every failure gets a short note in the test doc: what failed, the cause, what I did, and whether the result is still usable. That note stops a false number from resurfacing in a roadmap argument six months later.

I also banned one sentence from my own vocabulary: "the test was significant." Significant is not valid. I would rather report an inconclusive test I trust than a win I cannot explain.

My Pre-Read Ritual, Summed Up

Once I triage a failed check, the ritual itself stays fixed. I run it in this order every time, no skipping.

  1. Check the traffic split first. Start on day one and log counts daily, because a sample ratio mismatch is a fever, not a diagnosis, and root cause hunting is slow: KDD 2019 paper on SRM.

  2. Confirm every key event fires at the rate you assumed. If the rate gaps between variants, treat it as a bug before you treat it as an effect.

  3. Sweep for overlapping tests on the same surface. Keep one meaningful change per user at a time.

  4. Freeze metric definitions and filters before launch. Timestamp any change, and if drift happened, split the data at the change point.

  5. Only then read the outcome. An untrustworthy result is worse than no result, because it ships and then gets quoted internally for a year.

Conclusion

An untrustworthy A/B test result is worse than no result, because it ships and then gets quoted for a year. Silent failures rarely announce themselves; they show up as a skewed traffic split, tracking rates that do not match assumptions, overlapping tests on the same surface, or metric definitions that drift mid flight. My pre-read ritual catches those problems before I ever look at the outcome, and every failed check gets a short note in the test doc so a false number cannot resurface later. I also stopped calling tests "significant." I want results I can explain, even when they are inconclusive.

Your next step is simple: before you read your next test, run the five checks in order and start with daily traffic split counts. Trust is the metric that compounds. Build it one clean test at a time.