Sample Ratio Mismatch (SRM) is a statistically significant difference observed between the actual proportions of sizes in control and treatment groups compared to what was initially planned for an experiment.
People conducting online controlled experiments, such as A/B testing, read this information when they need to diagnose potential failures or inconsistencies within their test group data.
External context
Detecting SRM suggests that the integrity of the experiment may be compromised because the observed ratios deviate significantly from the expected traffic split. This mismatch often signals a technical issue related either to the randomization process itself or to problems with the instrumentation used during the controlled testing environment.
Sample ratio mismatch Wikipedia contributors, “Sample ratio mismatch”, en.wikipedia.orgLicence01What it is and how it works
SRM operates on the principle of expected proportions versus observed proportions. In a controlled experiment, you define an initial split (e.g., 1:1). If your system is working correctly, the final counts for key metrics—like brand mentions or click-through rates—should roughly maintain that original ratio across both groups. When SRM flags an issue, it means the observed data distribution has shifted away from the expected baseline. This shift suggests a systematic error, such as unequal exposure to the test variant, improper logging of user actions, or differential traffic routing between the control and treatment groups. It is not merely random variation; it points toward a structural failure in how the experiment was implemented or measured.
Simply put, SRM is an alarm bell for data analysis. If you send 50% of users to Group A and 50% to Group B, but when you look at the results, Group A seems to have done 65% of the action while Group B only did 35%, that's a Sample Ratio Mismatch. It means something interfered with your test setup.
02What to do about SRM
If you detect an SRM warning, your immediate action must be caution. Do not trust the primary metric results until the root cause is found. First, check the experiment's logging infrastructure. Verify that the tracking mechanism correctly attributed all user interactions to their intended group (Control or Variant). Next, review the traffic allocation logs directly; look for evidence of users being incorrectly bucketed or routed outside the defined test parameters. If the mismatch persists after checking the setup, consider pausing the analysis and consulting with your data engineering team to validate the integrity of the underlying data pipeline before proceeding.
- Review traffic logs for unexpected drop-offs or surges in specific groups. — warn
- Confirm that all metrics (e.g., impressions, clicks) are logged uniformly across both test arms. — check
03How it is measured or noticed
Detection involves comparing the actual observed counts against the expected counts, which are calculated by multiplying the total number of observations in a group by its assigned proportion. For example, if 10,000 users were split 50/50, you expect 5,000 interactions from each side. If your system reports only 4,200 interactions for one side and 5,800 for the other, that difference is flagged by SRM. The statistical test quantifies how unlikely this observed imbalance is if the experiment were truly random and executed perfectly according to plan.
How the record puts it
In the design of experiments, a sample ratio mismatch (SRM) is a statistically significant difference between the expected and actual ratios of the sizes of treatment and control groups in an experiment.
04Common Mistakes Regarding SRM
Misunderstanding SRM can lead marketers to incorrectly discard valid test results or overreact to minor statistical noise. It is crucial to understand that an SRM warning does not automatically mean your brand visibility strategy failed; it means the measurement of that strategy might be flawed.
- Assuming a small, persistent mismatch is just 'noise.' A consistent deviation indicates a systematic failure point in tracking or routing. — warn
- Ignoring the initial traffic split documentation. Always cross-reference the observed totals against the planned distribution ratios. — check
05Limitations and Confusion Points
SRM is a powerful tool for data integrity, but it has boundaries. It primarily detects issues related to distribution or exposure, not necessarily the underlying causality. For instance, if your test was intentionally designed with unequal traffic splits (e.g., 90/10 because of external platform constraints), SRM will flag a 'mismatch' against an assumed 50/50 split—but this is expected behavior based on the design. Furthermore, it cannot account for external market shifts or changes in user search intent that occur mid-test; those are confounding variables requiring separate analysis.
06Worked Example of Mismatch
Consider a test where you compare two AI search result layouts. You plan to send 10,000 users equally (5,000 Control, 5,000 Variant). If the final logged data shows that only 4,800 users were successfully tracked in the Control group and 5,200 in the Variant group, this represents a clear SRM. The discrepancy is not random; it means the tracking system failed to capture an expected number of impressions for one side.
If the planned split was 50/50 (Expected: 5,000 / 5,000), but the observed counts are 4,800 / 5,200, an SRM is flagged due to the imbalance in recorded sample size.
The entry above is written by GetLoopLoop. What follows is what independent catalogues hold about the same term — none of it is the source of this page.
The same term on Wikipedia
Catalogued in 2 languagesFrequently asked questions
How is SRM different from standard statistical significance testing?
SRM focuses specifically on data integrity and traffic allocation rather than just result variability. Standard significance tests check if an observed difference between groups (like conversion rates) is likely due to chance, while SRM checks if the input proportions of users into those groups were maintained as expected.
If I see a minor SRM warning, should I stop my test immediately?
No, seeing a minor SRM warning does not necessitate an immediate halt to your testing. It signals caution and suggests investigating the potential source of the deviation—such as tracking errors or unexpected traffic shifts—before invalidating valid results.
What is the practical threshold for considering an SRM mismatch severe enough to invalidate my data?
There is no universal, fixed percentage threshold; severity depends on your overall experiment goals and sample size. Generally, a deviation that suggests a major imbalance in how traffic was split across groups should prompt deep investigation into the tracking mechanism itself.
Does SRM only apply to metrics like clicks or impressions, or can it flag issues with other outcomes?
While SRM is often calculated using easily quantifiable counts like clicks and impressions, its underlying principle applies to any measured outcome. It signals that the ratio of users expected to generate a specific outcome did not match the actual observed ratio.
If I am running multiple tests simultaneously, can SRM help me isolate which test is causing data leakage or imbalance?
Yes, by monitoring SRM across different segments and groups, you can pinpoint exactly where the expected proportions are failing. This allows you to narrow down whether the issue lies with a specific treatment group's implementation or the overall traffic routing mechanism.
Wikimedia Commons
Related visuals with source and licence credit
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
You need to check for sample ratio mismatch. This warning means that the actual distribution of users across your test and control groups has significantly deviated from what was planned. You must first pause analysis and immediately investigate your tracking setup or traffic allocation mechanism.
It sounds like you need to check for sample ratio mismatch. This indicates that the observed outcomes are not proportional to the traffic splits expected at the start of the experiment. The fix is usually technical—revisiting how your tracking codes or randomization system is firing.
You need to confirm whether there's a sample ratio mismatch first. This means the discrepancy could be due to an issue with data collection or traffic routing, rather than poor performance of your creative. Always verify the integrity of the input data before drawing conclusions.