Sample size

Also called: statistical power

How many users an experiment needs to detect an effect of a given size with a given probability, called power (usually 80%). It is decided before the test, from the smallest effect worth detecting: the minimum detectable effect (MDE).

Formulan per group ≈ (1.96 + 0.84)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ Δ²

Try it · 30 seconds · from Atterna’s practice

Day 3 of a fixed-horizon test planned for at least 14 days and a target sample size: the variant is +8%, p = 0.03.

Ship it?

Preparing the practice…

How we work it through

1
Pick the smallest effect worth acting on.
2
Compute the sample it needs before starting.
3
Stop at that sample, not at the first good-looking day.

On the practice dataset

The query we run on Meridian, Atterna’s practice marketplace, and what it returns.

-- users per group to detect +1 point on a 10% base,
-- 95% confidence, 80% power
SELECT POWER(1.96 + 0.8416, 2)
       * (0.1002*0.8998 + 0.1102*0.8898) / POWER(0.01, 2);

Resultabout 14,800 per group — the badge test has about 5,800

The common trap

Running until “it looks significant” instead of until the planned sample size is reached.

When an AI answers

An AI assistant

“Two days of data are enough, the result is already significant.”

What it didn’t know: It was not told the planned sample size or that daily checks inflate false positives.
Ask back: “What sample size was planned, and has the test reached it?”

Paste a real AI answer into the checker — it writes these questions for you →