A/B test

An experiment that randomly splits users into groups that see different versions, then compares a pre-registered metric between them.

Formularandom split → one metric fixed in advance → compare at the planned end

Try it · 30 seconds · from Atterna’s practice

Day 3 of a fixed-horizon test planned for at least 14 days and a target sample size: the variant is +8%, p = 0.03.

Ship it?

Preparing the practice…

How we work it through

1
Check the split before the result.
2
Read the metric and the end date fixed before the start.
3
Say what the test cannot tell: long-term effects, other segments.

On the practice dataset

The query we run on Meridian, Atterna’s practice marketplace, and what it returns.

SELECT a.variant, COUNT(DISTINCT a.user_id) AS users,
       COUNT(DISTINCT o.user_id) AS buyers
FROM meridian.ab_assignments a
LEFT JOIN meridian.ab_orders o ON o.user_id = a.user_id
GROUP BY a.variant;

Resultcontrol 599 of 5,978 (10.0%), badge 666 of 5,663 (11.8%) — first check the split

The common trap

Stopping as soon as the difference looks significant, or checking many metrics and reporting the one that moved.

When an AI answers

An AI assistant

“The badge lifts conversion by 17%, ship it.”

What it didn’t know: It did not check that the groups have the sizes the design promised.
Ask back: “Is the split 50/50 as planned? Run a sample ratio check first.”

Paste a real AI answer into the checker — it writes these questions for you →