Every experimentation platform you have ever used is solving one embarrassing problem: people look at the results before the test is over, and looking changes the answer. The fix has a forbidding name — the sequential probability ratio test — and a very simple idea underneath it.
01 · The problem
Looking early is not cheating. It just breaks the math you were using.
You ship a new checkout flow. You want to know if it beats the old one. So you split traffic, and because you are a human being with a dashboard, you check on Monday. And Tuesday. And Wednesday. The moment it crosses the significance line, you call it and move on.
That feels like diligence. It is actually a slot machine. A test result wanders up and down as visitors trickle in, and the more often you look, the more chances you give it to wander across the line by luck alone.
Here is what that costs. Below, the two checkout flows are secretly identical — there is no winner to find. Every "winner" the simulation declares is a false alarm.
Run an A/A test — two identical flows — and choose how often you peek
The dotted lines are the classic 95% significance boundaries. The test statistic wanders because traffic is random. Peeking marks (◆) are the moments you check.
Two identical flows, checked repeatedly. With a single look at the end, false alarms sit at the 5% you asked for. At twenty looks, roughly a quarter of these tests crown a winner that does not exist. Nothing about the data changed — only how often you asked.
The usual advice is "stop peeking — wait for the full sample." That advice loses. Waiting means burning traffic on a losing variant for weeks, and nobody with a business actually obeys it. The better answer is to use a test that expects to be watched continuously.
02 · The idea
Put the evidence on a scale and agree in advance how far it has to tip
The fix is to stop thinking about sample size at all. Instead: weigh the evidence after every single observation, and set two lines in advance. One line means "enough — the new flow wins." The other means "enough — it doesn't." Between them, you simply keep collecting.
Three zones, decided before a single visitor arrives
Nothing about this changes when you look at it, because "how far it has tipped" is the only thing that ever decides.
The two outer lines are chosen from the error rates you are willing to accept: a 5% chance of a false winner and an 80% chance of catching a real one. They are fixed before the test starts and never move — which is exactly why checking a thousand times is harmless.
03 · The mechanism
Every visitor is a tiny vote, and the votes multiply
What actually sits on the scale is a ratio. After each visitor you ask one question: which story explains what I just saw better? Story A says the conversion rate is the old 10%. Story B says it is the 12% you were hoping for.
If the visitor buys, story B looks better — it predicted buying more often. If the visitor leaves, story A looks slightly better. You multiply those verdicts together as they arrive, and the running product is your evidence.
One visitor moves the needle by a lot, or by almost nothing
The asymmetry is the whole engine: conversions are rare, so a conversion is loud and a non-conversion is a whisper.
Because 88% and 90% are nearly the same number, a non-conversion is almost no evidence at all. It takes roughly eight non-buyers to cancel out one buyer. That imbalance is why these tests can resolve faster than you would guess.
That formula is easier to trust once you have watched it tick. Below is the exact worked example from the original version of this piece — four visitors, one at a time, with the arithmetic shown at every step. Press Next visitor and follow the number.
Walk the ledger one visitor at a time and watch Λ get multiplied
Each visitor multiplies the running evidence by 1.2 if they buy and by 0.978 if they leave. Nothing else happens.
The first four visitors are the scripted example — buy, leave, leave, buy. After that the ledger keeps drawing real visitors from a flow that genuinely converts at 12%, so you can watch the walk all the way to a boundary.
Four visitors move the evidence from 1.000 to 1.377 — nowhere near the 16 it needs. That is the honest shape of this test: a conversion is worth 20% more evidence, a non-conversion costs barely 2%, and it takes about eight non-buyers to cancel a single buyer. To clear the upper boundary at all you need roughly twenty more conversions than a 10% flow would have produced. Sequential testing is not magic. It just refuses to waste traffic once the answer is already in.
Now let the same machine run at full speed. Drag the slider to set what is actually true — something the real world never tells you — and let visitors arrive one at a time.
Set the true conversion rate, then watch the evidence walk until it hits a line
The test is always asking the same question — is it 10% or 12%? — no matter what you set the truth to.
Set the truth to 10% and the line drifts down: no lift to find, and the test says so. Set it to 12% and it climbs, but slowly — small effects are genuinely hard. Set it to 16% and it slams into the ceiling in a few hundred visitors. The test spends exactly as much traffic as the question requires.
04 · The payoff
The same error rates, at roughly half the traffic
This is the part that makes companies adopt it. A conventional fixed-sample test for this exact question — 10% baseline, a 20% relative lift worth detecting, 5% false alarms, 80% power — demands about 1,471 visitors before you are allowed to look. The sequential version reaches the same error guarantees far sooner, because most experiments are not close calls.
Traffic needed to reach the same 5% / 80% guarantees
Measured over 6,000 simulated experiments each. The sequential test's cost varies because it stops when it is done.
A 36% saving when the new flow really is better — and a 53% saving when it isn't, which is the case that matters most. Losing ideas get killed in half the time, and that traffic goes to the next idea instead.
05 · The catch
To run this test you must name the exact effect you are looking for
Look again at the evidence formula. It needs p₁ — one specific number. Not "some improvement." Not "anything better than today." A single committed guess, made before you have seen any data, about how big the lift will be.
That is a strange thing to be sure about. And getting it wrong is not a rounding error.
The truth is a 12% conversion rate. Here is what your guess cost you.
Power is the chance of correctly detecting the real lift. Each cell is 3,000 simulated experiments.
Aim at a lift far bigger than the real one and the test confidently concludes "no win" — fast, and wrong. It isn't broken. You asked it whether the lift was 20%, it correctly answered no, and you heard "the new flow doesn't work."
06 · The extension
mSPRT: stop betting on one number and spread the bet across a range
The mixture sequential probability ratio test changes one thing. Instead of a single alternative story, you carry several at once — a 12% story, a 15% story, a 20% story — each with a weight saying how plausible you find it. The evidence ratio becomes a weighted blend.
One committed guess versus a spread of them
The mixture is just a shape over "how big might the lift be" — a spike if you are certain, a spread if you are not.
Three components with equal weights is the simplest useful mixture. You can also use two, or a smooth curve over every lift between 0% and 30% — the shape is yours to choose, and it encodes what you actually believe before the test starts.
Here is the honest accounting — and it is more interesting than "the mixture is better."
Nobody knows the true lift in advance. So compare strategies across every truth they might face.
Each cell: power (chance of catching a real lift) and the median number of visitors it took. 3,000 simulated experiments per cell.
| Your strategy | Truth is 12%a modest lift | Truth is 15%a solid lift | Truth is 20%a huge lift | Worst casewhat you risk |
|---|---|---|---|---|
| Guess 12% | 0.81760 visitors | 1.00312 | 1.00145 | 0.81 |
| Guess 15% | 0.30141 | 0.81134 | 0.9971 | 0.30 |
| Guess 20% | 0.1234 | 0.3943 | 0.8238 | 0.12 |
| Mixture of all three | 0.49244 | 0.93172 | 1.0065 | 0.49 |
Read down the last column. The mixture is never the best strategy in any single column — but it never collapses either. That is what buying insurance looks like on a spreadsheet.
And there is a finding in that table worth stating plainly, because it is the practical advice most write-ups skip: for a simple one-sided test like this one, guessing the smallest lift you would care about beats the mixture. Aim at 12% and a real 20% lift still slams into the ceiling — faster, in fact. Aiming low is already robust.
So why does the mixture exist? Because it pays off precisely where that trick stops working:
- When the alternatives don't sit on one line. "Aim low" only makes sense if bigger effects are strictly easier to detect. Two-sided tests, multiple variants, or effects that could go either way have no "low" to aim at.
- When you want one test valid at every effect size and every stopping time. Mixing over a continuum of alternatives is what produces the always-valid p-values and confidence sequences behind commercial "peek whenever you like" dashboards.
- When you genuinely hold a prior. If past experiments say lifts cluster around 3%, encoding that beats pretending you know it is exactly 3%.
07 · The connection
Widen the mixture far enough and you have reinvented the Bayes factor
If you have met Bayesian statistics, that integral over a weighted range of alternatives should look familiar. It is a Bayes factor: the evidence for one hypothesis over another, averaged across the parameter values you think are plausible.
The same machine, seen from two directions
How wide you spread the weight is the only dial that separates them.
Squeeze a Bayesian sequential test's prior down to a single point and its Bayes factor becomes the SPRT's likelihood ratio — the same number, computed the same way. Widen it, and the two answers separate.
There is still a real philosophical difference underneath the shared arithmetic. The SPRT holds both of its stories fixed and only asks which one the data favours; it never learns a posterior. A Bayesian sequential test keeps updating its belief about the effect size as evidence arrives. In practice, with an informative prior, the two behave almost identically — which is why arguments about this are usually less consequential than they sound.
08 · In practice
What to actually do on Monday
- If you peek, you need a sequential test. Not more discipline. A test whose guarantees survive being watched.
- Name the smallest lift worth shipping — not the one you are hoping for. It is the robust choice, and it is also the only number your business can actually defend.
- Expect variable duration. Sequential tests finish when they finish. Budget traffic, not calendar days, and set a hard cap so an inconclusive test cannot run forever.
- Reach for the mixture when the effect could go either way, when several variants compete, or when you want one dashboard that is valid at every glance.
- Don't stack it with other peeking. Sequential testing buys you one specific freedom — continuous monitoring of a pre-declared metric. It does not license hunting through twenty metrics and ten segments for something that crossed a line.
The deepest idea here is small and portable, and it long outlives A/B testing: decide in advance how much evidence would change your mind, then watch the evidence and stop when it does. Most of the statistical trouble in experimentation comes from running that sentence backwards — watching first, and deciding what counts as convincing afterwards.
All figures in this piece come from simulation rather than closed-form approximation: 3,000–6,000 synthetic experiments per cell, with a 10% baseline conversion rate, α = 0.05 and 80% power. The stopping boundaries are Wald's A = (1−β)/α = 16 and B = β/(1−α) ≈ 0.21. Simulated false-alarm and power rates matched their targets to within one percentage point, which is the check that the boundaries are doing what they claim.