Post-hoc analysis refers to statistical procedures conducted after data has been collected in a study, usually to explore specific significant differences among multiple independent groups.
This topic is essential for researchers conducting scientific studies who are interpreting results derived from analyses of variance (ANOVA).
External context
For those writing about research methodology, understanding post-hoc analysis means knowing that these statistical checks happen after the data has been observed. While an initial test like ANOVA can confirm if differences exist among three or more groups, a separate post-hoc procedure is required to identify precisely which specific groups are responsible for those detected variations.
Post hoc analysis Wikipedia contributors, “Post hoc analysis”, en.wikipedia.orgLicence01What it is and how it works
It starts with a hypothesis that wasn’t part of the original plan. You take the same dataset you used for the main test, split it into groups, and run additional tests—like t‑tests or chi‑square—on those groups. The key is that you’re not changing the data, just exploring more angles.
Post‑hoc analysis means looking at data after the main test to spot extra patterns.
02What to do about it
First, decide which extra questions are most business‑relevant. Then, use the same statistical software you used for the main test. Run the tests, record the p‑values, and compare them to your chosen alpha level. If the p‑value is below 0.05, you can report a significant finding, but note that it’s post‑hoc and may need replication.
03How it is measured or noticed
Look for a list of additional tests in your analytics dashboard. In Google Search Console, the Search Analytics report can be filtered by query, page, country, and device. If you see a new segment that shows a statistically significant lift in clicks or impressions, that’s a post‑hoc signal.
How the record puts it
In a scientific study, post hoc analysis consists of statistical analyses that were specified after the data were seen.
04Common mistakes
- Running tests on the same data you used for the main analysis can inflate Type I error.
- Treating post‑hoc findings as definitive without replication.
- Ignoring the need to adjust the significance threshold for multiple comparisons.
- Failing to document the hypothesis before running the test.
05Limits
Post‑hoc analysis is not a substitute for a pre‑registered study. It doesn’t replace a planned experiment and can’t prove causality. It’s also easy to confuse it with exploratory data analysis, which is broader and often not tied to a specific hypothesis.
06Worked example
"After seeing the main results, we noticed that the mobile‑device segment had a 12% higher click‑through rate. Running a post‑hoc t‑test on that segment confirmed the difference (p = 0.03)."
The entry above is written by GetLoopLoop. What follows is what independent catalogues hold about the same term — none of it is the source of this page.
The same term on Wikipedia
Catalogued in 7 languagesFrequently asked questions
How is post-hoc analysis different from a pre-planned analysis?
Post-hoc analysis is conducted after data collection, using hypotheses that were not defined beforehand, whereas pre-planned analysis follows a pre-registered protocol. It can reveal patterns that were overlooked but is more prone to bias.
When should I consider doing a post-hoc analysis?
You should consider it when initial results are inconclusive or when you suspect additional insights might exist, but only if the data set is large enough to support further testing. However, always treat findings as exploratory.
How do I actually perform a post-hoc analysis?
First, identify the new questions that matter to your business, then run the corresponding statistical tests in your analytics dashboard. Make sure to document the process to differentiate it from confirmatory tests.
Is a post-hoc analysis still valid if it finds a significant effect?
The significance can be real, but because the hypothesis was generated after seeing the data, the result is considered exploratory and should be confirmed in a new study.
What can go wrong if I rely too heavily on post-hoc analysis?
Over-reliance can lead to false positives, misleading stakeholders, and wasted resources on initiatives that were not truly supported by robust evidence.
How long after data collection can I run a post-hoc analysis before it becomes unreliable?
There is no hard cutoff, but the sooner you analyze, the less chance that the data has drifted or been altered; still, the findings should be treated as hypothesis-generating rather than definitive.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Usually, you should run a quick post-hoc check to see if any unexpected trends appear. Look for any significant deviations in the engagement metrics and compare them against your baseline. If the drop is consistent across multiple metrics, it’s worth investigating further.
It depends—if the drop is due to a sudden change in search algorithms, a post-hoc analysis can help identify the cause. Run the analysis on the last quarter’s data to spot any new patterns. Then explain that the findings are exploratory and will guide next steps.
Yes, you can perform a post-hoc analysis right from the dashboard. Start by selecting the time period of the dip and run the additional tests for brand mentions and search volume. The results will show whether the dip correlates with competitor activity or a platform change.