A/B testing is a user experience research method that uses randomized experiments to compare multiple versions of a single variable, determining which variant achieves the most effective outcome.
This topic is typically read by digital marketers or UX designers who are researching ways to improve conversion rates and often consult it alongside guides on statistical hypothesis testing.
External context
For individuals working on optimizing web pages, A/B testing provides a structured way to conduct controlled experiments by comparing different versions of content or design elements. By applying principles of statistical analysis, users can scientifically determine if one variant is significantly more effective than another. This process ensures that decisions about functionality are based on measurable data rather than assumptions.
A/B testing Wikipedia contributors, “A/B testing”, en.wikipedia.orgLicence01What it is and how it works
A/B testing works by randomly splitting your audience into two groups. Group A sees the control version (the current page or element). Group B sees the variant (the change you want to test). You then measure a predefined success metric — such as click-through rate, conversion rate, or engagement time — for each group. The experiment runs until you collect enough data to reach statistical significance, which tells you whether the observed difference is likely real or due to chance. This method isolates the effect of a single change, so you can attribute any difference in performance to that variable. Google Search Central recommends running tests for at least one full business cycle to account for weekly patterns.
A/B testing shows you which of two options works better by showing each to a random half of your audience and measuring the result.
02What to do about it
Start by forming a clear hypothesis: 'If I change X, then Y will improve by Z%.' Choose one variable to test — for example, the headline, call-to-action button color, or image. Set up the experiment using a testing tool or by manually splitting traffic. Run the test for a sufficient duration, typically at least one to two weeks, to gather enough data. Avoid peeking at results early; wait until the sample size reaches the required threshold for statistical significance. Once the test concludes, analyze the results and implement the winning variant if the improvement is significant and practical. Document your findings to inform future tests.
03How it is measured or noticed
The primary measurement is the conversion rate for each variant, calculated as the number of desired actions divided by total visitors. You also track the lift — the percentage difference between the two rates. Statistical significance is usually set at a 95% confidence level, meaning there is only a 5% probability that the difference is due to random chance. Tools like Google Optimize or custom scripts report these metrics. To notice if A/B testing is happening on a site, look for URL parameters like '?variant=B' or experiment tags in the page source. Some platforms also display a banner or cookie indicating active experiments.
How the record puts it
A/B testing is a user-experience research method.
04Common mistakes
- Testing too many variables at once — this makes it impossible to know which change caused the result.
- Stopping the test as soon as one variant appears to be winning — early data is often unreliable.
- Running the test for too short a period — you need to capture full user behavior cycles, including weekends.
- Ignoring statistical significance — a difference that looks big may still be due to chance.
- Not randomizing traffic properly — bias can skew results.
05Limits
A/B testing works best when you have enough traffic to reach statistical significance within a reasonable time. For low-traffic sites, the required sample size may take months, making the test impractical. It also assumes that the only difference between groups is the variable being tested; external factors like seasonality or marketing campaigns can confound results. A/B testing is not suitable for testing multiple variables simultaneously — for that, use multivariate testing. It is also often confused with personalization, which tailors content to individual users rather than comparing two fixed versions. Finally, ethical considerations arise when testing affects user experience negatively; always have a plan to revert if the variant harms performance.
06Worked example
A SaaS company wants to increase sign-ups on its pricing page. Hypothesis: Changing the call-to-action button from 'Start Free Trial' to 'Get Started Free' will increase click-through rate by 10%. They set up an A/B test with 50% of visitors seeing the original button (A) and 50% seeing the new button (B). After two weeks, 1,000 visitors saw A with 50 sign-ups (5% conversion) and 1,000 visitors saw B with 65 sign-ups (6.5% conversion). The lift is 30%, and the p-value is 0.03, below the 0.05 threshold. The test is statistically significant. The company implements the new button and sees a sustained increase in sign-ups.
The entry above is written by GetLoopLoop. What follows is what independent catalogues hold about the same term — none of it is the source of this page.
- Also called
- A-B testing, A/B test
- Kind of thing
- type of test
The same term on Wikipedia
Catalogued in 24 languagesFrequently asked questions
How is A/B testing different from multivariate testing?
A/B testing compares two versions of a single variable, while multivariate testing tests multiple variables simultaneously to find the best combination. A/B testing is simpler and requires less traffic to reach statistical significance. Use A/B testing when you have a clear hypothesis about one change, and multivariate testing when you need to optimize several elements at once.
Should I always run an A/B test before making a change?
No, only when the change is significant enough to warrant the effort and you have enough traffic to reach statistical significance in a reasonable time. For minor tweaks with low impact, a test may not be worth the delay. Prioritize tests on high-traffic pages or changes that directly affect conversion goals.
How do I determine the sample size needed for an A/B test?
Use a sample size calculator based on your baseline conversion rate, the minimum effect you want to detect, and your desired statistical power. A common rule is to aim for at least 1,000 conversions per variant, but the exact number depends on your parameters. Running the test until you have enough data prevents premature conclusions.
Does A/B testing still work if I have low traffic?
It can, but you need to run the test much longer to reach statistical significance, and you may only be able to detect large effects. With very low traffic, the risk of inconclusive or misleading results increases. Consider using Bayesian methods or sequential testing to make better use of limited data.
What happens if I stop an A/B test too early?
You risk a false positive — concluding that one variant is better when the difference is actually due to random chance. Early stopping inflates the error rate and undermines the reliability of your decision. Always wait until the test reaches the pre-calculated sample size or a clear stopping rule.
How long should I run an A/B test?
Run the test until you reach statistical significance, but also account for full business cycles (e.g., weekdays and weekends) to avoid day-of-week effects. A minimum of one to two weeks is common, but low-traffic tests may take months. Avoid peeking at results and stopping early based on temporary fluctuations.
Wikimedia Commons
Related visuals with source and licence credit
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Run a controlled experiment where you show the old headline to half your visitors and the new one to the other half, then compare the conversion rates. This way you can be confident that any difference is due to the change, not random variation. Make sure you have enough traffic to reach a statistically valid conclusion.
It depends on how big a difference you expect. With low traffic, you can only detect large effects reliably, and the test may need to run for weeks or months. If the expected improvement is small, you might need to accept a higher risk of inconclusive results or use a different method like a Bayesian analysis.
Check the p-value and confidence interval — if the p-value is above 0.05 or the interval includes zero, the result is not statistically significant. Do not stop the test early; continue until you reach the planned sample size. If you already stopped, treat the result as inconclusive and consider running a follow-up test.