Sample ratio mismatch (SRM)

When observed group sizes are statistically unusual under an experiment’s planned split, commonly checked with chi-square against a pre-set alert threshold. It is a warning to investigate assignment and logging before interpreting the metric, not proof of a bug.

Formulaχ² of observed group sizes against the planned split

Try it · 30 seconds · from Atterna’s practice

A planned 50/50 test recorded 5,978 users in control and 5,663 with the badge. The split check gives p ≈ 0.004; the SRM alert threshold was set at 0.01 before the test.

The dashboard says the badge won. What should you do first?

Preparing the practice…

How we work it through

1
Compare the split-check p-value with the alert threshold fixed before the test.
2
If alerted, check assignment, logging and segments; the p-value alone does not identify a cause.
3
Resolve the data-quality concern before interpreting the metric.

On the practice dataset

The query we run on Meridian, Atterna’s practice marketplace, and what it returns.

SELECT variant, COUNT(*)
FROM meridian.ab_assignments
GROUP BY variant;

Result5,978 control and 5,663 badge — p ≈ 0.004 for a 50/50 design

The common trap

Reading the result anyway.

ExampleMeridian’s delivery-badge test assigned 5,978 users to control and 5,663 to the badge. For a 50/50 design that gap has a chi-square p-value of about 0.004. With a pre-set SRM threshold of 0.01 it triggers an investigation; the count alone does not identify a cause.

When an AI answers

An AI assistant

“The groups are 51/49, close enough to 50/50.”

What it didn’t know: It judged by eye; these counts give p ≈ 0.004 under the planned 50/50 split, below a 0.01 alert threshold.
Ask back: “What was the pre-set SRM threshold, and what do assignment, logging and segment checks show?”

Paste a real AI answer into the checker — it writes these questions for you →