Blog
Statistical Power and Sample Size: Why Tiny Peptide Studies Mislead
Statistics is where many peptide papers quietly fail. Not because of fraud, but because the experiments were too small to tell signal from noise. A few simple concepts let you spot the problem in minutes.
Key takeaways
- Power is the chance a study detects a real effect.
- Effect size is how big the difference is relative to the spread of the data.
- Small samples miss real effects and exaggerate the ones they find.
- Look for sample size justification, correction for multiple tests and independent replicates.
The four linked quantities
- Alpha, the false-positive rate, conventionally 0.05.
- Power, conventionally 80 percent, meaning a 20 percent chance of missing a real effect.
- Effect size, often Cohen’s d: the difference between group means divided by the standard deviation.
- Sample size per group.
Fix any three and the fourth follows. Most researchers fix alpha and power, estimate effect size from pilot data, and calculate the sample size.
Numbers to remember
For comparing two groups with a t-test at alpha 0.05 and 80 percent power, a quick rule is n per group of about 16 divided by d squared. That gives roughly:
- d = 2 (very large effect): about 6 per group.
- d = 1 (large): about 17 per group.
- d = 0.5 (medium): about 64 per group.
- d = 0.2 (small): about 400 per group.
So a group size of three can reliably detect only a gigantic difference, roughly three standard deviations. If a study with three animals per group finds a smaller effect and calls it significant, treat it with caution.
The winner’s curse
When power is low, the effects that cross the significance line tend to be the ones that were overestimated by chance. The published number is then biased upward, and replication studies find smaller effects. This helps explain why preclinical findings so often shrink. A widely cited 2012 report from Amgen scientists said they could confirm the findings of only 6 of 53 landmark preclinical cancer papers.
P-values in plain words
A p-value of 0.03 does not mean a 97 percent chance that the effect is real. It means that, if the compound did nothing at all, data this extreme would turn up about 3 percent of the time. It says nothing about the size or importance of the effect. Confidence intervals show both magnitude and uncertainty and are more informative.
Multiple comparisons
Test 20 outcomes at alpha 0.05 and you expect about one false positive by chance. Studies that measure many endpoints and highlight those that moved are describing noise. Corrections such as Bonferroni, or predefined primary outcomes, protect against this.
Pseudoreplication
Ten wells from one culture dish, or ten measurements from one animal, are not ten independent samples. The unit of replication is the independent biological unit: the animal, or an independent experiment run on a different day. Treating technical replicates as independent inflates certainty.
A reader’s checklist
- Is there a stated sample size and a reason for it?
- Were groups randomized and outcomes assessed blind?
- Was the primary outcome declared in advance?
- Do the figures show individual data points, not just bar graphs?
- Are the replicates biological or technical?
The bottom line
Good statistics does not make a result true, but bad statistics makes any result untrustworthy. Pair these checks with the controls in our controls guide and the ladder in the evidence ladder.
Related reading
- Vehicle, Scrambled and Positive Controls in Peptide Studies
- The Evidence Ladder: How to Read Peptide Research Without Being Fooled
- Why Self-Experiments Mislead: Placebo, Regression to the Mean and the n-of-1
- Browse the full Research Library
For laboratory research use only. Not for human or veterinary use, and not intended to diagnose, treat, cure or prevent any condition. This article is educational and is not medical advice.