S3U

Reading / Media literacy / Graduate / ree01

Audit Missing Evidence in a Research Synthesis

Construct a sensitivity analysis for selective reporting.

All worksheets
5 questions0m 0s

Prerequisites

Research protocols, study versus report units, effect estimates, and missing-data mechanisms.

Learn the skill

Begin with a prespecified question and a study-level inventory. Compare registrations and protocols with reports. Selectively unavailable results can distort a synthesis; a symmetrical plot or nonsignificant diagnostic does not prove their absence.

Worked example

Three reports from one cohort are not three independent studies. A registered outcome missing from a report is a reason to investigate, not evidence that its effect was zero.

Model and Assumptions

An invented registry lists six independent studies of one educational outcome. Four reported study estimates are 1, 2, 3, and 6. Two estimates are unavailable. For a deliberately simple sensitivity exercise, weight all six studies equally and let each unavailable estimate lie between -4 and 4. This is not an inverse-variance meta-analysis.

1. What is the sum of the four observed estimates?
2. What is the smallest six-study average under the stated bounds?
3. What is the largest six-study average under those bounds?
4. Does this bounded exercise yield a confidence interval?
5. What would most directly investigate selective reporting?

Further inquiry

Derive bounds for the six-study average when missing estimates instead lie in [a,b]. Discuss how the conclusions depend on those bounds, equal weighting, and the possibility of unregistered studies.

Review criteria

  • Show the interval [(12+2a)/6, (12+2b)/6].
  • Do not equate a sensitivity range with sampling uncertainty.
  • Define a reproducible study-level search and discrepancy log.