Statistical significance

Also called: p-value

A p-value is the probability of seeing a difference at least as large as the observed one if there were no real effect. “Significant” means it is below a threshold chosen before the test, often 0.05.

Formulap = chance of a difference at least this large if there were no effect

Try it · 30 seconds · from Atterna’s practice

Day 3 of a fixed-horizon test planned for at least 14 days and a target sample size: the variant is +8%, p = 0.03.

Ship it?

Preparing the practice…

How we work it through

1
Fix the threshold and the end date before the test starts.
2
Compute the p-value once, at the planned end.
3
Report the size of the effect with its interval, not just “significant”.

On the practice dataset

The query we run on Meridian, Atterna’s practice marketplace, and what it returns.

-- two-proportion z-test on the badge test
-- control 599/5,978 = 10.02%, badge 666/5,663 = 11.76%
SELECT (0.1176 - 0.1002) / SQRT(0.1002*0.8998/5978
                                + 0.1176*0.8824/5663) AS z;

Resultz ≈ 3.0, p ≈ 0.003 — significant, and still unusable while the split is broken

The common trap

Reading p = 0.03 as “97% sure the variant works”, treating significance as size or importance, or peeking daily and stopping on the first significant day.

When an AI answers

An AI assistant

“p = 0.003, the badge definitely works.”

What it didn’t know: It did not know the split failed its sample ratio check.
Ask back: “Is the assignment sound? Significance assumes a fair split.”

Paste a real AI answer into the checker — it writes these questions for you →