How long should you run an A/B test? Sample size, duration and the checks before you read it

“Is the test significant yet?” is the wrong first question. An A/B test runs for as long as it takes to detect the smallest effect worth acting on, and you decide that before it starts. Stopping a fixed-horizon test when the numbers first look good inflates the false-positive risk.

Short version: pick the minimum effect worth detecting, estimate the sample size from your baseline rate, divide by daily traffic and round up to whole weeks. Before reading any result, check that the groups are the size the design promised. Investigate statistically unexpected imbalances before interpreting the effect; a random split will rarely produce exactly equal counts.

The three inputs

1. Baseline rate — the current conversion of the metric you test. Example: 10% of users in a period place an order.
2. Minimum detectable effect (MDE) — the smallest change that would change your decision. Detecting a 20% relative lift is cheap; detecting 5% is expensive.
3. Traffic — how many users enter the test per day.

With the usual settings (5% significance, 80% power, two groups of equal size), the users you need per group for a 10% baseline:

The numbers come from the standard two-proportion formula; any online sample-size calculator gives similar results. Halving the effect you want to detect roughly quadruples the sample.

Round up to whole weeks

Weekdays and weekends behave differently. A test that runs Monday to Thursday measures a different population from one that runs a full week. For a business with a weekly cycle, plan complete weeks. Also allow for conversion delays and the time needed to reach your sample target; one week is a useful planning heuristic, not a universal statistical minimum.

Before reading the result: the sample ratio check

On the Meridian practice marketplace, a “fast delivery” badge test assigned users 50/50 over 13 days. The result looks great: 11.8% of users in the badge group ordered against 10.0% in control, a z-score of about 3, “highly significant”.

But the groups have 5,663 and 5,978 users. For a 50/50 design, a gap that large has a chi-square p-value of about 0.004. Whether this triggers a sample ratio mismatch (SRM) alert depends on a threshold chosen before looking at the results. It would trigger at 0.01 but not at 0.0005, the conservative threshold described by Microsoft’s experimentation team. The imbalance is a reason to investigate the assignment and logging, not proof of a particular bug. Do not choose a threshold afterwards to validate a preferred conclusion.

How to look for the cause:

• Split the group sizes by device, platform, app version, country and day. An assignment or logging issue may be concentrated in one segment — a release, a redirect, a caching layer.
• Check whether users can switch groups, or are counted twice after logging in on a second device.
• Check whether one variant loads slower or fails more often, so fewer of its users are logged.

One episode of the Meridian simulator is built on exactly this test, so we won’t spoil where the split breaks.

Check your own group sizes in the free SRM checker. It runs in your browser; a warning alone does not prove a bug.

Four mistakes that make a test lie

1. Peeking. Checking every day and stopping at the first p < 0.05 inflates false positives far above 5%. Fix the duration in advance, or use a method designed for continuous monitoring.
2. Choosing the metric afterwards. Testing many metrics increases the chance of a false positive unless you account for multiple comparisons. Name the primary metric before the start.
3. The novelty effect. Users click a new badge because it is new. Plan how to examine changes over time; report the full planned window and any exploratory period breakdowns, rather than discarding early results after seeing them.
4. Ignoring the split. Always run the SRM check first. It takes a minute.

Ask your AI assistant

Assistants happily compute a p-value from two conversion rates. Ask them instead: “What were the planned and actual group sizes, and is there a sample ratio mismatch? Was the duration fixed in advance? What is the confidence interval of the effect, not just the p-value?” The free AI Answer Checker adds the questions about group sizes and a fixed duration automatically when an answer mentions an experiment.

FAQ

How long should an A/B test run?

Plan for the sample size needed to detect your minimum meaningful effect and the relevant business cycle and conversion delay. For weekly patterns, round up to full weeks. Compute the sample size from the baseline rate and the effect before the test starts; don’t stop early because the result looks significant.

What is a good minimum detectable effect?

The smallest change that would alter your decision, given the cost of the change. For many product tests that is a 5–20% relative lift. Smaller effects need much larger samples: halving the effect roughly quadruples the users you need.

What is a sample ratio mismatch?

A statistically unexpected difference between observed group sizes and the planned allocation. Evaluate it against a predefined threshold. It can signal assignment, logging or analysis problems; the test flags a symptom, not the cause, and random imbalances are possible. See the glossary.

Can I stop an A/B test early if it is significant?

Not with a standard fixed-sample test: repeated peeking makes false positives much more likely than the 5% you think you have. Either fix the duration in advance or use a sequential method built for early stopping.

Where can I practise analysing A/B tests?

In one episode of the Meridian simulator, a product manager asks whether the delivery badge worked and how long to wait; you check the split and the result and end with a decision backed by your numbers. Meridian is part of Atterna Analyst (€59) and each episode opens after the previous one; the free level check shows where to start today.