How peer review works, and what it can and cannot catch

TUREFF• •0

News stories often describe a finding as "peer-reviewed" as if that settles the matter. Peer review is valuable, but it is a quality filter run by busy volunteers, not a guarantee that a result is true. Knowing how it works helps you judge how much weight to give a paper.

What happens to a paper after submission

The authors submit a manuscript to a journal, usually with the data, methods and a cover letter explaining why the work matters.
An editor screens it. Many papers are rejected at this stage, without review, because they are out of scope, too weak or clearly flawed.
The editor invites reviewers, typically two or three researchers who work in the same field. They are usually unpaid and do the work alongside their own research.
Reviewers write reports. They comment on whether the question is worthwhile, whether the methods can answer it, whether the conclusions follow from the results, and what is missing.
The editor decides: accept, minor revisions, major revisions or reject. Most accepted papers go through at least one round of revision, and sometimes several.
The paper is published, and from then on the wider community can examine it.

The main kinds of peer review

Single-blind: reviewers know who the authors are, but authors do not know who reviewed them. This is still common.
Double-blind: neither side knows the other's identity, which aims to reduce bias based on reputation, institution or gender. In small fields reviewers can often guess anyway.
Open review: names and sometimes the reports themselves are published alongside the paper, which makes the process more accountable.
Post-publication review: comments, critiques and replication attempts after a paper appears. In practice this is where many serious problems are found.

What peer review is good at catching

Obvious methodological gaps, such as a missing control group or a measurement that cannot detect what the authors claim.
Overclaiming, where the conclusions go further than the data. Reviewers frequently ask authors to soften their language.
Missing context, such as earlier studies that found something different.
Unclear writing that would stop other scientists from understanding or repeating the work.
Statistical choices that need justifying, especially when a journal uses a specialist statistical reviewer.

What it usually cannot catch

Fabricated or manipulated data. Reviewers normally see summaries and figures, not the raw notebooks or instrument files. A determined fraudster can produce numbers that look plausible.
Errors buried in code or spreadsheets that the reviewers were never sent or did not have time to rerun.
Selective reporting, such as experiments that failed and were quietly left out, or outcomes that were swapped after the results came in. Pre-registration, where a study's plan is recorded in advance, exists partly to tackle this.
Whether the result will replicate. A single well-designed study can still be a fluke. Only repeating it can show that.
Reviewer blind spots. Two or three people cannot be expert in every technique a modern paper uses.

Preprints, retractions and corrections

A preprint is a paper posted publicly before peer review, common in physics, biology and medicine. Preprints speed up sharing, but they have not been checked by a journal, so treat their claims as provisional, and be wary of news stories that do not mention this. When serious problems surface after publication, a journal may issue a correction for a fixable error or a retraction when the findings can no longer be relied upon. A retraction is a sign that the system eventually worked, not proof that all science is broken, though it does show that publication alone is not the end of scrutiny.

Predatory journals

Some outlets charge authors a fee and publish almost anything with little or no real review. Warning signs include unsolicited emails inviting submissions, promises of very fast acceptance, vague editorial boards and a name that imitates a well-known journal. A paper that appears only in such a venue deserves extra caution.

How to use this as a reader

Treat "peer-reviewed" as a minimum, not a seal of truth.
Ask whether other independent groups have found the same thing.
Check whether the paper has been corrected or retracted; journals usually mark this on the article page.
Notice whether a news story is reporting a preprint, a conference talk or a published paper.

Have you ever followed a paper from preprint to publication, or reviewed one yourself? Share your experience in the comments.