Confidence interval

A range of values for the true effect that the data are consistent with. At 95% confidence, the method that builds it captures the true value in 95 of 100 repeated experiments. A wide interval means the test cannot tell you much yet.

Formulaestimate ± 1.96 × standard error (95%)

How we work it through

1
Report the interval next to every estimate.
2
Ask whether the whole interval is worth acting on.
3
A wide interval means “run longer”, not “it worked”.

On the practice dataset

The query we run on Meridian, Atterna’s practice marketplace, and what it returns.

SELECT 0.0174 - 1.96 * 0.00578 AS low,
       0.0174 + 1.96 * 0.00578 AS high;
-- difference in buyer rate, badge − control

Result+1.7 points, 95% interval +0.6 to +2.9

The common trap

Reporting only the point estimate (“+6%”) when the interval runs from −2% to +14%.

When an AI answers

An AI assistant

“The badge adds exactly 1.74 points of conversion.”

What it didn’t know: It reported the point estimate as a fact.
Ask back: “What is the 95% interval, and would its low end still justify shipping?”

Paste a real AI answer into the checker — it writes these questions for you →