Sample Size Calculator for A/B Testing
Determine how many visitors you need per variant to reliably detect a change in conversion rate. Calculate required sample size for conversion-rate experiments with statistical power analysis. Works offline, no signup required.
Your current or expected conversion rate for the control variant (between 0.1% and 99.9%).
A 20% relative lift on 5% baseline means 6% conversion in the variant group (5% × 1.20).
Power of 0.80 (80%) is standard: an 80% chance of detecting your specified effect if it truly exists. Higher power requires larger samples.
α = 0.05 is standard. Smaller α means fewer false positives but larger sample size.
If provided, we estimate how many days the test needs to run. Used only for arithmetic; does not model visitor overlap or repeat users.
§2The formula
For a two-proportion z-test, the sample size per variant is:
- n
- Sample size per variant (control and treatment each require this many visitors)
- z₍α/₂₎
- Critical z-value for significance level α, two-tailed (e.g., 1.96 for α = 0.05)
- z₍β₎
- Critical z-value for power: normInv(power), e.g., 0.842 for power = 0.80
- p₁
- Baseline conversion rate (control group)
- p₂
- Expected conversion rate under the alternative (treatment group)
- |p₁ − p₂|
- Absolute difference in proportions (the effect size)
§3How to compute p₂ from MDE
The calculator converts your MDE input into p₂:
- Relative % lift: If you specify a 20% relative lift and p₁ = 5%, then p₂ = 5% × (1 + 0.20) = 6%.
- Absolute percentage points: If you specify +1.5 pp and p₁ = 5%, then p₂ = 5% + 1.5% = 6.5%. The calculator clamps p₂ to (0, 1) to avoid impossible values.
Both methods are equally valid; choose based on what is meaningful for your business context.
§4Reference: sample sizes across common scenarios
Below is a table of required sample sizes n per variant for a range of baseline conversion rates and relative MDEs, all at power = 80% and α = 0.05 (two-tailed):
| Baseline rate | +10% lift | +20% lift | +30% lift |
|---|---|---|---|
| 2% | 80,679 | 21,106 | 9,795 |
| 5% | 31,231 | 8,155 | 3,778 |
| 10% | 14,749 | 3,839 | 1,772 |
| 20% | 6,507 | 1,680 | 769 |
§5Key assumptions and caveats
The peeking problem
If you check your results before collecting the full sample size—or run multiple independent tests without correcting for multiple comparisons—you inflate the false-positive rate (Type I error). The α = 0.05 assumes you test once at the end. Each interim analysis multiplies your actual error rate. Use a sequential testing framework or Bonferroni correction if you must peek.
Independence of observations
The formula assumes each visitor is independent: one visitor's outcome does not influence another's, and no visitor is counted twice. If the same user visits both variants or returns multiple times, they violate independence. This inflates both Type I and Type II error rates. If repeat users are common in your product, inflate your sample size estimate by a design effect (typically 1.3–2.0).
Intention-to-treat versus per-protocol
The sample size assumes that all assigned visitors complete the test. If some users drop out or leave before converting, you need larger samples. Add a dropout/attrition buffer (e.g., 10–20%) to your final sample size.
Fixed sample size, not sequential
This calculator assumes you decide on n, run the test, then analyze once. Sequential designs (testing after every N visitors) have different error rates and require specialized software.
§6FAQ
How many visitors do I need for my A/B test?
It depends on your baseline conversion rate, the minimum effect you want to detect, your desired power, and your significance level. Smaller effects and lower baseline rates require much larger samples. Use this calculator: enter your baseline rate, specify the effect size (as a relative % lift or absolute percentage points), choose power (usually 80%), and significance (usually 95%, α = 0.05). The calculator returns the sample size per variant.
What is a minimum detectable effect (MDE)?
The MDE is the smallest change in conversion rate you care about detecting. It represents the effect size you want the test to be powered to find. If your baseline is 5% and you specify a 20% relative lift, your MDE is 1 percentage point (5% → 6%). Smaller MDEs require larger samples. Choose an MDE based on business impact: if a 0.5% lift is worth the engineering effort, set that as your MDE.
Why do lower baseline conversion rates need much bigger samples?
Conversion rates near 0% or 100% have very little variance, but rates near 50% have maximum variance. The formula depends on p(1−p), the variance of a Bernoulli trial. At p = 2%, the variance is 0.02 × 0.98 = 0.0196. At p = 50%, it is 0.5 × 0.5 = 0.25—13 times larger. To detect a relative effect when baseline variance is low, you need many more observations.
Should I use relative % lift or absolute percentage points?
Both are valid; choose based on your business context. A relative % lift (e.g., 20% lift on 5% baseline) is often more interpretable for UX decisions: "we increased conversions by a fifth." An absolute percentage point (pp) gain (e.g., +1 pp) is clearer if you want to commit to a fixed business outcome. Relative lifts require larger samples for low baseline rates, because the absolute difference is smaller. Either way, choose an MDE you can justify to your stakeholders.
What happens if I check results before reaching the pre-computed sample size?
You inflate your false-positive rate. The significance level α = 0.05 is a guarantee only if you run the test once to completion and analyze once. Peeking—checking intermediate results and deciding whether to stop early—turns your real α into something much larger (sometimes 0.15 or higher, depending on how many times you peek). If you must monitor progress, use a sequential testing framework with corrected critical values, or plan to correct for multiple comparisons (e.g., Bonferroni: divide α by the number of planned interim analyses).
§7Related calculators
See also:
- Sample Size Calculator (hub) — Compare means, proportions, and survey margin of error.
- Sample Size Calculator for Experiments — Two-means power calculation for continuous outcomes.
- Sample Size Calculator for Surveys — Single-proportion confidence intervals for survey design.
- Effect Size Calculator — Convert between Cohen's d, r, η², and odds ratio.
- Z-Test Calculator — Hypothesis test for proportions and means.
§8Sources
- Kohavi, R., & Longbotham, R. (2017). "Online Controlled Experiments and A/B Testing." In Encyclopedia of Machine Learning and Data Mining. Springer. — Definitive reference on A/B testing methodology and power calculations in practice.
- NIST/SEMATECH e-Handbook of Statistical Methods. "Hypothesis Tests for a Single Proportion" and "Hypothesis Tests for the Difference of Two Proportions." — Detailed formulas and examples for two-proportion tests.
- UCLA OARC: Sample Size Determination — Tutorial on power analysis and sample size for various designs.
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum. — Foundational reference for statistical power and conventions.