Bias is systematic error that can distort a study's findings. It differs from random variation: increasing the sample size does not necessarily remove it. A precise estimate can still be misleading if the comparison, measurements, missing data, or reporting process is flawed.
Bias assessment should identify a concrete mechanism and the result it could affect. A vague statement that a study is “biased” is less useful than explaining how participants were selected or how an outcome was measured differently between groups.
Follow the path from recruitment to reporting
Selection can affect who enters a study or remains available for analysis. Measurement can differ because groups receive different questions, observers know the intervention, or records vary in completeness. Missing outcome data can distort a comparison when the reason for missingness relates to the result being measured.
Selective reporting can occur when investigators present only favorable outcomes, time points, or analyses. Comparing a report with its protocol or registration can help identify unexplained changes, although an unexplained change needs investigation rather than an automatic allegation of misconduct.
An illustrative missing-data problem
Suppose a training study reports high satisfaction among people who completed all sessions. Participants who found the schedule unmanageable stopped attending and were not asked for feedback. The reported satisfaction may describe completers accurately while giving an incomplete view of everyone offered the training.
The appropriate question is not simply whether the sample was large. It is whether the missing experiences could change the conclusion about acceptability. This hypothetical example does not establish how any real program performs.
A better reading note would identify who was excluded, what information is missing, and which conclusion is affected. It should not invent the missing participants' responses.
Separate risk from demonstrated distortion
A method may create a risk of bias without revealing its exact size or direction. If outcome assessors knew assignments, explain why that knowledge might influence the particular outcome. An objective laboratory measure and an impression-based rating can raise different concerns.
Cochrane's randomized-trial framework evaluates specific results across domains, rather than treating an entire paper as uniformly trustworthy or untrustworthy. Different outcomes in the same study may have different measurement or missing-data problems.
Use appraisal to improve the decision
A checklist is a prompt for reasoning, not a score that replaces judgment. Record the evidence supporting each concern and whether it could materially alter the interpretation. Avoid assuming that peer review, a famous institution, or a sophisticated analysis removes all methodological weaknesses.
When studies disagree, compare their methods as well as their conclusions. An apparently inconsistent literature may include important differences in populations, outcome definitions, and bias risks.
See confounding in health research for a common problem in observational comparisons, and systematic reviews for nurses for how appraisal informs a synthesis.
When participation itself may affect the result, start with how the sample was selected.
Sources
- Cochrane Handbook, Chapter 8: Assessing risk of bias in a randomized trial (accessed 2026-09-17).
- CDC Field Epidemiology Manual: Collecting Data (accessed 2026-09-17).