A/B Test Sample Size & Significance Calculator
Calculate the exact sample size needed for your experiment or test live split test results for 95% statistical significance.
Test Planning Parameters
~25 Days
Technical Deep-Dive: A/B Testing Statistics Explained
๐ On this page
- 1. Why sample-size planning has to come before a test, not after
- 2. Statistical power and why 80% is the standard target
- 3. Understanding the Z-score and p-value
- 4. Relative vs. absolute Minimum Detectable Effect
- 5. Common mistakes that invalidate real A/B tests
- 6. What this calculator does not account for
1. Why sample-size planning has to come before a test, not after
The single most common mistake in conversion-rate optimization is launching an A/B test without first calculating how much traffic it needs, then eyeballing results as they trickle in and calling a winner the moment the numbers look good. This approach systematically produces false positives, because small early samples are naturally noisy โ a coin flipped 10 times can easily land 7 heads by pure chance, but the same coin flipped 10,000 times will land extremely close to 50/50. Pre-calculating the required sample size before launch, using your current baseline conversion rate and the smallest improvement worth detecting, is what turns an A/B test from a guessing exercise into a real statistical experiment.
2. Statistical power and why 80% is the standard target
Statistical power is the probability that your test correctly detects a real effect when one actually exists โ the flip side of significance, which guards against false positives. An underpowered test (below roughly 80% power) can easily miss a genuinely better variant and report an inconclusive result even though the variant truly does convert better; you simply didn't collect enough data to see it clearly. 80% power paired with a 95% significance threshold (ฮฑ = 0.05) is the widely adopted industry default that this calculator's sample-size formula is built around, balancing test duration against reliability.
3. Understanding the Z-score and p-value
The Z-score measures how many standard deviations apart the two observed conversion rates are, given the amount of data collected โ a small Z-score means the gap could easily be noise, while a large Z-score means the gap is unlikely to have occurred by chance. The p-value converts that Z-score into a more intuitive probability: specifically, the probability of seeing a difference this large (or larger) purely by chance if there were actually no real difference between Control and Variant. A p-value below 0.05 (equivalent to the 95% confidence level shown by this calculator) is the conventional cutoff for calling a result statistically significant.
4. Relative vs. absolute Minimum Detectable Effect
This calculator's MDE input is a relative percentage, not an absolute one โ entering "15" with a 3% baseline conversion rate means you want to detect a jump to roughly 3.45% (a 15% relative improvement over 3%), not a jump to 18%. This distinction matters enormously for sample-size math: relative MDE is the standard convention in most CRO tooling because it scales sensibly regardless of whether your baseline conversion rate is 1% or 20%, but it is easy to misread if you are expecting an absolute-percentage-point input.
5. Common mistakes that invalidate real A/B tests
Beyond stopping early, the most frequent errors that quietly invalidate an otherwise well-run test include: running the test for less than one full business cycle (missing weekday/weekend conversion-rate variation), changing the variant mid-test after launch (which mixes data collected under two different conditions), running multiple simultaneous tests that interact with the same page or funnel, and failing to account for returning visitors who see both variants across sessions due to cookie/cache issues. Any of these can produce a confident-looking but ultimately misleading result.
6. What this calculator does not account for
This tool implements the standard fixed-horizon two-proportion test โ it assumes a single binary conversion goal, a fixed pre-determined sample size, and independent visitor sessions. It does not implement Bayesian testing, sequential/always-valid testing methodologies (which allow safe continuous monitoring), multi-armed bandit allocation, or multiple-comparison correction for tests with more than two variants or more than one success metric. For advanced experimentation programs running many concurrent tests, a dedicated experimentation platform with built-in guardrails against these more complex statistical pitfalls is worth the investment.
Frequently Asked Questions
What is statistical significance in A/B split testing?
Statistical significance measures the likelihood that a difference in conversion rates between Control and Variant is real, rather than a result of random chance. The industry standard threshold is 95% confidence (p < 0.05).
Why should an A/B test run for a minimum duration even if it reaches 95% significance early?
Ending tests early exposes experiments to "peeking errors" and false positives caused by day-of-week seasonality. Tests should generally run for at least 1 to 2 full business cycles (7-14 days).
What is Minimum Detectable Effect (MDE)?
MDE is the smallest relative percentage improvement in conversion rate you want your experiment to be able to reliably detect. Smaller MDEs require substantially larger sample sizes.
Why does a smaller Minimum Detectable Effect require a much larger sample size?
Detecting a subtle difference between two conversion rates requires more data to separate a real signal from ordinary day-to-day noise, the same way distinguishing two very similar-sounding notes requires more careful listening than distinguishing two obviously different ones. The relationship is not linear either โ the required sample size scales roughly with the inverse square of the MDE, so halving the effect you want to detect roughly quadruples the sample size needed.
What do all the numbers on this page actually run on โ client-side math or a server?
Every calculation on this page runs instantly in your browser using standard statistical formulas: the sample-size planner uses the widely cited two-sample binomial approximation (16ยทpฬ(1-pฬ)/ฮดยฒ, attributable to Evan Miller's public sample-size methodology, calibrated for a 5% significance level and 80% statistical power), and the significance tester uses a two-proportion Z-test with a standard normal-CDF approximation to derive the p-value. Nothing is sent to a server and no external API is called โ your experiment data never leaves your device.
Is a 95% confidence result guaranteed to be a permanently correct decision?
No. A 95% confidence level means that if you ran the same experiment repeatedly under identical conditions, you would expect a false positive roughly 1 time in 20. It is strong statistical evidence, not a mathematical certainty. Combining a proper pre-calculated sample size (so you are not underpowered), a fixed minimum test duration (so you are not peeking early), and a sensible business read of the result gives you the most reliable real-world decision.
Should I stop a test the moment it crosses 95% significance?
Generally no. Checking results continuously and stopping the instant a threshold is crossed ("peeking") inflates the real false-positive rate well above the nominal 5%, because you are effectively giving yourself many chances to hit a lucky streak. Best practice is to pre-calculate the required sample size and/or duration before starting, then check the result only once that predetermined threshold is reached โ or use a sequential-testing methodology specifically designed to allow safe early stopping.