The 5 statistical mistakes that quietly ruin scientific papers
Most problematic statistical analyses are not caused by fraud. They come from small methodological decisions made early in the project — and those decisions can completely change the interpretation of a scientific result.
Watch the full video
YOUTUBE_LINK_PLACEHOLDER
Every year, scientific papers are corrected, criticized after publication, or sometimes retracted. In many cases, the core issue is not misconduct. It is something more ordinary: a statistical error that should have been addressed before submission.
Below are five mistakes I repeatedly encounter in biological, medical, omics, and imaging studies. They are common, subtle, and often serious enough to undermine an otherwise well-designed project.
01 Pseudoreplication: when your N is not your N
Imagine you have data from three mice. From each mouse, you measure one hundred cells. You now have three hundred measurements.
The temptation is obvious: run a statistical test using N = 300. The problem is that these three hundred cells are not independent observations. Cells from the same animal share the same biological context, history, genotype, environment, and experimental handling.
In that situation, your effective biological sample size is closer to three than to three hundred. Treating all cells as independent artificially inflates statistical power and can produce extremely convincing p-values for effects that may not replicate.
If you measured 300 cells from 3 animals, your biological N is usually 3, not 300 (unless your model explicitly accounts for the nesting structure (mixed models, pseudobulk approaches, etc.).
A more appropriate strategy is often to use a hierarchical or mixed-effects model, where the experimental unit is explicitly represented.
# Linear mixed model with a random intercept for animal lmer(value ~ group + (1 | animal))
In plain language: the model estimates the effect of the group while accounting for the fact that several measurements come from the same animal.
If you are unsure whether your experimental design suffers from pseudoreplication, this is exactly the type of issue we investigate at Probabilityχ. A design review often takes less than an hour and can prevent major problems later during peer review.
02 Multiple testing and p-hacking
The more hypotheses you test, the more likely you are to obtain statistically significant results purely by chance.
This is particularly important in RNAseq, proteomics, metabolomics, high-content screening, biomarker discovery, and any analysis where hundreds or thousands of comparisons are performed at once.
With a significance threshold of p < 0.05, testing 1,000 independent null hypotheses would produce around 50 significant results by chance alone. Without correction, a results table can look biologically meaningful while being dominated by false positives.
This is why multiple-testing corrections exist, including Bonferroni, Holm, and Benjamini-Hochberg false discovery rate control.
Common correction strategies
- Bonferroni correction for strict family-wise error control
- Holm correction as a less conservative stepwise alternative
- Benjamini-Hochberg FDR for high-dimensional omics analyses
P-hacking emerges when researchers try multiple analysis strategies and report only the significant ones. It is not always intentional. Sometimes it is simply the result of exploring many reasonable options without documenting the full analytical path.
Choosing the correct correction depends heavily on your experimental design and data structure. One of the most common reviewer criticisms we encounter concerns inappropriate handling of multiple testing. Reviewing the Methods section before submission is often much easier than responding to statistical concerns after rejection.
03 Correlation is not causation
Two variables can move together without one causing the other. This sounds obvious, yet many papers still move from association to causal language too quickly.
For example, observing that a gene is associated with poor survival does not prove that this gene drives poor survival. It may be a marker of another biological process, a proxy for disease severity, or a consequence rather than a cause.
This problem is especially common in observational cohorts, transcriptomic studies, biomarker discovery, and machine-learning models trained on biological or clinical datasets.
Words such as “drives”, “causes”, “regulates”, or “mediates” require stronger evidence than a correlation coefficient or a survival association.
Establishing causality generally requires stronger designs, such as intervention experiments, randomized designs, perturbation studies, Mendelian randomization, or explicit causal assumptions formalized through frameworks such as directed acyclic graphs.
This is often where reviewers become particularly critical. If you are working with observational cohorts, transcriptomics, or biomarker studies and are unsure whether your interpretation overreaches the data, an external methodological review can be extremely valuable before submission.
04 Violating test assumptions
A statistical test is only valid under the assumptions that justify it. Many analyses go directly from data to p-value without checking whether those assumptions are even approximately satisfied.
Common problems include non-normal distributions, unequal variances, repeated measurements treated as independent, small sample sizes, zero-inflated count data, censored survival data, and strong outliers.
The result may look statistically significant while being statistically invalid.
Before trusting the p-value
- Are observations independent?
- Is the distribution compatible with the model?
- Are variances comparable across groups?
- Are sample sizes sufficient for the chosen method?
- Are repeated or hierarchical measurements modeled correctly?
A sophisticated test applied to the wrong data structure remains the wrong analysis.
We are preparing a more detailed guide on statistical assumptions and model diagnostics for biological datasets. If you are dealing with hierarchical, longitudinal, count, ordinal, or survival data, additional checks are often required beyond standard workflows.
05 Statistical significance is not biological significance
A p-value tells you how compatible your data are with a null hypothesis. It does not tell you whether the observed effect matters biologically, clinically, or scientifically.
With sufficiently large sample sizes, very small effects can become highly significant. A 1% change in a biomarker may produce p < 0.000001 while remaining practically irrelevant.
This is why effect sizes, confidence intervals, uncertainty, and domain knowledge are essential. The key question is not only whether an effect is statistically detectable, but whether it is meaningful.
“Statistically significant” does not mean “important”. It only means that the result is unlikely under a specific null model.
06 The meta-lesson
Most statistical problems originate long before the analysis itself. They emerge from experimental design, sampling strategy, model selection, and interpretation.
Statistics should not be treated as the final administrative step of a project. It is part of the scientific process from the beginning.
Before believing a p-value, ask how it was obtained.
Manuscript checklist
- Is the experimental unit clearly defined?
- Are technical and biological replicates separated?
- Are multiple comparisons corrected when needed?
- Are causal claims supported by causal evidence?
- Are model assumptions checked and reported?
- Are effect sizes discussed, not only p-values?
If one of these five issues sounds familiar because you are preparing a submission, answering reviewers, or starting a new project and want to build the analysis on solid ground, Probabilityχ offers a free 30-minute technical consultation.
No sales pitch. Just a technical discussion about your data, your design, and your questions.
Preparing a manuscript or answering statistical reviewer comments?
Let's look at the design before the statistics become a problem.