Free Marketing Sample Size Calculator

Why sample size matters at all

A/B tests work by detecting a difference between two variants that's larger than what random noise alone would produce. Without enough sample, real effects look like noise and noise looks like real effects. The sample size formula tells you how many visitors per variant you need before the math can reliably distinguish signal from noise at your chosen confidence and power thresholds.

The math, top to bottom

For a two-tailed proportions test:

n = (z_a + z_b)^2 * (p1*(1-p1) + p2*(1-p2)) / (p1 - p2)^2

where:
  p1 = baseline conversion rate
  p2 = p1 * (1 + mde)        // relative MDE
  z_a = critical value at confidence level (1.96 for 95%)
  z_b = critical value at power (0.84 for 80%)

The numerator (z_a + z_b)^2 grows with confidence and power. The denominator (p1 - p2)^2 shrinks rapidly as MDE shrinks, which is why detecting tiny lifts requires huge samples. Doubling MDE roughly quarters the sample size. Halving the baseline conversion rate roughly doubles the sample size.

Why one-tailed tests are usually wrong

One-tailed tests assume you only care about detecting an effect in one direction. If the variant performs worse than the control, you treat that as "no effect" and move on. That's almost never how marketing tests actually run. You care about both wins and losses (a loss tells you to keep the control). Two-tailed tests are the honest default. They require roughly 25 percent more sample but produce a legitimate read on either direction of effect.

The peeking problem

The most common A/B testing mistake is checking p-values before the predetermined sample is reached. Each peek is essentially another statistical test, and the false positive rate inflates with every look. A test with a real 5% false positive rate becomes 20%+ if you peek 5 times. Either commit to checking exactly once at the predetermined sample, or use methods designed for sequential testing (Bayesian, or frequentist with O'Brien-Fleming style alpha spending).

Realistic test horizons

• Landing page CTA test, 5% baseline, 10% MDE: ~31K per variant. At 1K daily traffic, ~62 days.
• Pricing page test, 8% baseline, 10% MDE: ~17K per variant. At 500 daily traffic, ~70 days.
• Email subject line, 30% open baseline, 5% rel MDE: ~30K per variant. Email volume usually allows 1-week tests.
• Ad creative test, 2% CTR baseline, 20% MDE: ~5K per variant. Easy at any decent ad spend.
• Onboarding flow, 60% step-completion, 5% MDE: ~2.5K per variant. Fast for high-traffic SaaS.

When the sample size is impossibly large

If the calculator returns 100,000+ per variant and you only get 200 visitors a day, the test is not feasible at your chosen MDE. Three options: increase MDE (often the right answer because you usually wouldn't act on a 2% lift anyway), pool tests across higher-traffic segments, or use Bayesian methods which can stop earlier when the effect is clearly large. Don't run an underpowered test and pretend the result is meaningful.

A worked example

Your landing page converts at 5 percent. You want to detect a 10 percent relative lift (5.0 to 5.5). At 95 percent confidence and 80 percent power, the formula gives you n ≈ 31,012 per variant, 62,024 total. At 1,000 daily visitors split evenly between control and variant, the test runs ~62 days. If 62 days is too long, drop MDE to 20 percent (detect 5.0 to 6.0): sample shrinks to ~7,800 per variant, test runs ~16 days. Same data, different question, very different feasibility.