SML AGENTIC GROWTH LOOPS FREE CORE · PRO ON DUAL · x402 API

Holdout test sample size calculator

Per group, n = (zα/2·√(2p̄(1−p̄)) + zβ·√(p₁(1−p₁)+p₂(1−p₂)))² ÷ (p₂−p₁)². Example: a 5% baseline conversion rate, detecting a lift to 6% at 95% confidence and 80% power, needs 8,158 users per group. At 2,000 users a day that is 9 days, rounded up to a 14-day run.

Updated · Script Master Labs

Calculator

Optional readout (your results)
p₂ = p₁ × (1 + lift)          r = test ÷ control          p̄ = (p₁ + r·p₂) ÷ (1 + r)
n_control = [z_α/2·√(p̄(1−p̄)(1+1/r)) + z_β·√(p₁(1−p₁) + p₂(1−p₂)/r)]² ÷ (p₂ − p₁)²
n_test = r × n_control        run = ⌈total ÷ daily users⌉, rounded up to whole weeks (min 7 days)
readout: pooled two-proportion z-test, two-sided p-value, unpooled CI on the difference

How to do it

  1. Pick the baseline rate. Use the conversion rate of the group that will not see the change.
  2. Choose the smallest lift worth detecting. Smaller lifts need far more users: sample size scales with 1 ÷ (difference)².
  3. Set confidence and power. 95% confidence and 80% power are common defaults.
  4. Run full weeks, then read out once. Enter control and test results to get the p-value, CI and decision.

Questions

Why run a holdout instead of trusting platform conversions?

Platforms count conversions that would have happened anyway. A randomized holdout (a control group that sees no ads or no change) measures the incremental lift your spend actually causes.

Why round the run length to whole weeks?

Weekday and weekend behaviour differ. Running full weeks samples each day equally so the result is not biased by when the test started.

What if my result is not significant?

Treat it as no detectable effect at that sample size. Do not stop early when it looks good: peeking inflates false positives.

Can I use an uneven split like 90/10?

Yes. Set the test share; the calculator uses the ratio r. Uneven splits need more total users for the same power.

Free here · Pro on Dual

Card or 21 crypto options at checkout on SML Dual Market. Code ALIGN15 takes 15% off on Dual.