A p-value represents the probability of obtaining test results at least as extreme as those actually observed, assuming that the null hypothesis being tested is correct.
Individuals reading about this topic are typically engaged in quantitative research or statistics, often encountering it when reviewing academic publications and statistical testing methods.
External context
Although reporting p-values is a common practice across many quantitative fields, users must be aware that the misinterpretation and misuse of these values are widespread issues. When writing about statistical results, it is crucial to accurately explain what a small p-value implies regarding the null hypothesis without oversimplifying its meaning.
P-value Wikipedia contributors, “P-value”, en.wikipedia.orgLicence01What it is and how it works
The p‑value is calculated from the test statistic of a statistical test. It represents the probability that the observed data, or something more extreme, would occur if the null hypothesis were true. A small p‑value indicates that the observed effect is unlikely under the null hypothesis, suggesting the null may be false.
It tells how likely your result would happen if nothing real was happening.
02What to do about it
If your p‑value is below a chosen threshold (commonly 0.05), you can reject the null hypothesis and consider the result statistically significant. For marketing, use a low p‑value to justify changes in campaigns or content. If the p‑value is high, keep the current strategy or collect more data.
03How it is measured or noticed
You see a p‑value in the output of statistical software or A/B testing dashboards. It is usually displayed as a decimal (e.g., 0.032) or a percentage (3.2%). In Search Console experiments, the p‑value appears next to the lift metric.
How the record puts it
In null-hypothesis significance testing, the p-value is the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct.
04Common mistakes
- Interpreting a p‑value as the probability that the null hypothesis is true.
- Assuming a p‑value below 0.05 proves causation.
- Ignoring the context of the test and sample size.
- Using the same p‑value threshold for all tests without considering power.
05Limits
P‑values do not measure the size of an effect or its practical importance. They also depend on sample size; very large samples can produce tiny p‑values for trivial effects. A p‑value is not a direct measure of confidence in a result, and it can be misused if the null hypothesis is poorly defined.
06Worked example
Suppose you run an A/B test on two landing pages. Page A gets 1,200 visits with a conversion rate of 4.5%, while Page B gets 1,200 visits with a conversion rate of 5.2%. Using a two‑sample proportion test, the calculated p‑value is 0.037. Because 0.037 < 0.05, you can conclude that Page B’s higher conversion rate is statistically significant and consider switching to it.
If the p‑value is 0.037, the result is statistically significant at the 5% level.
The entry above is written by GetLoopLoop. What follows is what independent catalogues hold about the same term — none of it is the source of this page.
- Kind of thing
- function
The same term on Wikipedia
Catalogued in 41 languagesFrequently asked questions
Is a p‑value the same as the probability that my hypothesis is true?
No, a p‑value is the probability of observing data at least as extreme as yours assuming the null hypothesis is true; it does not give the probability that the hypothesis itself is true.
Should I rely solely on the p‑value to decide if my A/B test result is meaningful?
Usually, you should consider the p‑value alongside effect size and business context; a low p‑value alone does not guarantee a meaningful change.
How do you calculate a p‑value in an A/B test?
You compute a test statistic, such as a t‑statistic or z‑score, then determine the probability of that statistic under the null; most analytics tools provide this automatically.
Did the concept of p‑values change after the reproducibility crisis?
The definition of a p‑value hasn't changed, but its interpretation and the emphasis on it have been debated; many now recommend reporting confidence intervals and effect sizes alongside.
What happens if I misinterpret a p‑value and act on a false positive?
You may deploy a change that actually has no real effect, wasting resources and possibly damaging user experience; you would notice a lack of expected lift in subsequent data.
How long does it take before a p‑value becomes reliable after running an A/B test?
The p‑value stabilizes once you have enough samples; early estimates can be misleading, so it's best to wait until you reach the planned sample size.
Wikimedia Commons
Related visuals with source and licence credit
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Yes, if the p‑value is below your threshold, you can consider it statistically significant, but also check the effect size; otherwise, you might want to wait for more data.
It depends. If the p‑value is below your significance level, you can consider the result reliable, but also consider the business impact.
Usually, a low probability indicator below 0.05 suggests statistical significance, but you should also look at the effect size and confidence intervals to ensure it's meaningful.