Selection bias

When the groups you compare differ before any treatment, because of how people got into them, so the difference in outcomes is not the effect you are measuring.

Formulacompare groups that could have made the same choice

Try it · 30 seconds · from Atterna’s practice

Tomás · marketing

Orders with AUTUMN20 were 2.5× bigger than orders without it! Can I put “AUTUMN20 makes baskets bigger” into the December plan?

-- the query is correct
AUTUMN20 orders   €91.81 average
without the code  €36.60 average

The query is correct and the numbers are real. What do you check first?

Preparing the practice…

How we work it through

1
Ask how people got into each group.
2
Compare only people who could have been in either group (baskets from €30).
3
Say what the comparison cannot prove.

On the practice dataset

The query we run on Meridian, Atterna’s practice marketplace, and what it returns.

-- the promo week, 14–20 September
SELECT promo_code, AVG(items_total_eur)
FROM meridian.orders
WHERE status = 'paid'
  AND created_at >= '2026-09-14' AND created_at < '2026-09-21'
GROUP BY promo_code;

ResultAUTUMN20 €91.81 · no code €36.60 — 2.5×, and AUTUMN20 only worked on baskets from €30. From €30 in both groups: €91.81 against €86.86.

The common trap

Comparing people who chose something (used a code, opened an email) with people who did not.

ExampleMeridian case 1: in the promo week, orders with the AUTUMN20 code averaged €91.81 against €36.60 without it — 2.5× — because the code only worked on baskets from €30. Comparing baskets from €30 in both groups, the gap is about 6% — smaller than its margin of error.

When an AI answers

An AI assistant

“Orders with the promo code are 2.5× bigger, so the code grows baskets.”

What it didn’t know: It did not know the code required a €30 minimum basket.
Ask back: “Could everyone use the code? Compare orders eligible for it in both groups.”

Paste a real AI answer into the checker — it writes these questions for you →