Holdout test sample size calculator
Per group, n = (zα/2·√(2p̄(1−p̄)) + zβ·√(p₁(1−p₁)+p₂(1−p₂)))² ÷ (p₂−p₁)². Example: a 5% baseline conversion rate, detecting a lift to 6% at 95% confidence and 80% power, needs 8,158 users per group. At 2,000 users a day that is 9 days, rounded up to a 14-day run.
Updated · Script Master Labs
Calculator
p₂ = p₁ × (1 + lift) r = test ÷ control p̄ = (p₁ + r·p₂) ÷ (1 + r) n_control = [z_α/2·√(p̄(1−p̄)(1+1/r)) + z_β·√(p₁(1−p₁) + p₂(1−p₂)/r)]² ÷ (p₂ − p₁)² n_test = r × n_control run = ⌈total ÷ daily users⌉, rounded up to whole weeks (min 7 days) readout: pooled two-proportion z-test, two-sided p-value, unpooled CI on the difference
How to do it
- Pick the baseline rate. Use the conversion rate of the group that will not see the change.
- Choose the smallest lift worth detecting. Smaller lifts need far more users: sample size scales with 1 ÷ (difference)².
- Set confidence and power. 95% confidence and 80% power are common defaults.
- Run full weeks, then read out once. Enter control and test results to get the p-value, CI and decision.
Questions
Why run a holdout instead of trusting platform conversions?
Platforms count conversions that would have happened anyway. A randomized holdout (a control group that sees no ads or no change) measures the incremental lift your spend actually causes.
Why round the run length to whole weeks?
Weekday and weekend behaviour differ. Running full weeks samples each day equally so the result is not biased by when the test started.
What if my result is not significant?
Treat it as no detectable effect at that sample size. Do not stop early when it looks good: peeking inflates false positives.
Can I use an uneven split like 90/10?
Yes. Set the test share; the calculator uses the ratio r. Uneven splits need more total users for the same power.
Free here · Pro on Dual
Card or 21 crypto options at checkout on SML Dual Market. Code ALIGN15 takes 15% off on Dual.