How to read a scientific study: study types, sample size and p-values explained

TUREFF• •0

Headlines turn single studies into certainties: coffee prevents cancer, coffee causes cancer. The study behind the headline usually says something much more modest. You do not need a science degree to read research critically; you need to know where to look and which questions to ask.

Abstract versus full paper

The abstract is a short summary at the top. It is useful for deciding whether a paper is relevant, but it is written to present the findings in their best light and leaves out the limitations. Before trusting a claim, look at:

Methods: who or what was studied, how many, for how long and how things were measured.
Results: the actual numbers, not just the words "significant" or "associated".
Limitations: most good papers list them near the end of the discussion. Authors are often more cautious here than in the abstract.

If the paper is behind a paywall, many are freely available as author copies or through PubMed Central; searching the title often finds one.

Study types, from weaker to stronger evidence

Case reports and case series: descriptions of one or a few patients. Useful for spotting something new, but they cannot show cause.
Animal and lab studies: show a possible mechanism. Many results in mice or cells never translate to people.
Cross-sectional studies: a snapshot of a population at one time. They show patterns but not what came first.
Case-control studies: compare people with a condition to similar people without it, looking back at past exposures. Good for rare diseases, but memory and selection can skew results.
Cohort studies: follow a large group over time, comparing those with and without an exposure. Stronger, but still observational.
Randomised controlled trials (RCTs): participants are randomly assigned to a treatment or a comparison (often a placebo). Randomisation balances other differences between groups, so RCTs can show cause and effect. Blinding, where participants or researchers do not know who got what, makes them stronger still.
Systematic reviews and meta-analyses: gather all the studies on a question and combine them. Usually the best evidence available, but only as good as the studies they include.

This ladder is a rule of thumb. A large, well-run cohort study can be more convincing than a tiny, sloppy trial.

Sample size and effect size

Small studies are more easily swayed by chance; a surprising result from 20 people deserves less trust than one from 20,000. Equally, look at how big the effect is. "Doubles the risk" sounds alarming, but doubling a 1-in-10,000 risk to 2-in-10,000 matters far less than the headline suggests. Look for absolute risk (how many people out of 100 or 1,000) as well as relative risk.

Correlation is not causation

Observational studies find associations: people who do X have more Y. That does not mean X causes Y. Other explanations include:

Confounding: a third factor causes both. People who drink red wine may also be wealthier, and wealth affects health.
Reverse causation: Y causes X. People who feel unwell may stop exercising, not the other way round.
Chance: test enough things and some will line up by accident.

Good observational studies adjust for known confounders, but they can never rule out the unknown ones.

P-values in plain terms

A p-value answers a narrow question: if there were really no effect, how surprising would results like these be? A p-value below 0.05 is conventionally called "statistically significant". It does not mean there is a 95% chance the finding is true, and it says nothing about whether the effect is large or important. Large studies can find tiny, meaningless effects significant; small studies can miss real ones. Confidence intervals are more informative: they give a range of plausible values for the effect. A wide interval means the estimate is uncertain.

Funding and conflicts of interest

Papers should declare who funded the work and any financial interests of the authors. Industry funding does not make a study wrong, but it is a reason to read the methods carefully and to see whether independent studies agree.

Preprints and peer review

A preprint is a paper shared before peer review. Preprints speed up science, but they have not been checked by independent experts, and some change substantially or are never published. Peer review itself is a basic quality filter, not a guarantee. One study, however good, is rarely the final word; look for whether others have replicated it.

Quick checklist

What kind of study is it, and in whom (people, animals, cells)?
How many participants, and for how long?
How big is the effect, in absolute terms?
Could something else explain the result?
Who paid for it, and has it been peer reviewed and replicated?

Share a headline that turned out to oversell its study, or ask about a paper you are trying to read, in the comments.