A real example
In Meridian, Atterna’s practice marketplace, a “fast delivery” badge test was designed as 50/50. It assigned 5,978 users to control and 5,663 to the badge. The dashboard called the badge a winner — but the split has a chi-square p-value of about 0.004. Press “Try a real example” above to see it. The right answer to the team is “not yet”: first find out why fewer users landed in the badge group.
What to check when the split is off
- Redirects and page speed: a slower variant or an extra redirect loses users before they are logged.
- Bots and internal traffic: crawlers or staff counted in one group only.
- Assignment bugs: returning users re-assigned, or a cookie that resets.
- Eligibility after assignment: a filter applied to one variant but not the other.
- Logging and pipelines: events dropped for one group, or a variant that fires its exposure event later.
Deciding how long to run a test and when to read it is covered in how long to run an A/B test; the term itself is in the glossary. The full episode with the broken split is part of the Meridian simulator.
FAQ
What is a sample ratio mismatch?
A sample ratio mismatch (SRM) is when the group sizes of an experiment are statistically unusual under the planned split — for example 51/49 instead of 50/50 on 20,000 users (p ≈ 0.005), while 52/48 on 2,000 users can still be chance. It is a signal that assignment or logging may be broken, so any difference in results may come from who ended up where rather than from the change. It calls for an investigation, not a verdict.
How is SRM tested?
With a chi-square goodness-of-fit test: compare the observed number of users in each group with the number expected from the planned split. A small p-value means counts at least this inconsistent with the plan would be unusual under the planned assignment model. Choose the alert threshold before the test; this checker flags p < 0.01, while teams monitoring many tests may use a stricter threshold such as 0.001.
What causes a sample ratio mismatch?
Usually something between assignment and logging: a redirect or a slower page that loses users in one group, bots or crawlers counted in one variant, a bug that re-assigns returning users, eligibility rules applied after assignment, or data pipelines that drop events from one group.
Can I read the A/B test result if there is an SRM?
Investigate the alert before using the metric result. If assignment or logging is biased, a significant metric can be misleading. An SRM alert alone does not prove a bug: document what the checks found and whether the data is valid for the intended comparison.
Is this SRM calculator free?
Yes, with no account. It runs in your browser and nothing you enter is sent anywhere. It is made by Atterna, where you practise analyst work — including a broken A/B split — on a realistic marketplace.