Prerequisites
Confusion matrices, proportions, representative sampling, and digital source criticism.
Separate the historical concept from its searchable representation. Validate both retrieved and unretrieved pages. Precision measures the correctness of hits; recall measures recovery of relevant pages. A search hit rate alone can reflect OCR errors as well as historical usage.
Worked example
If a sample contains ten relevant pages and a search finds seven of them plus one irrelevant page, recall is 7/10 while precision is 7/8. These denominators answer different questions.
Model and Assumptions
Fictional random validation sample: 200 pages, 50 truly containing a target phrase and 150 not containing it. Search returns 35 true positives and 5 false positives, missing 15 relevant pages. Treat manual labels as correct for this exercise. The target is phrase occurrence, not endorsement of its meaning.
Further inquiry
Calculate specificity and derive p_observed=sensitivity*p+(1-specificity)*(1-p). Solve for p when the denominator is nonzero. Discuss uncertainty, transport to other periods, and why using the same validation sample to check its own correction is not independent validation.
Review criteria
- Obtain specificity 145/150 and corrected p=0.25 for the supplied sample rates.
- State the stable-error-rate and reference-label assumptions needed for transfer.
- Keep phrase detection distinct from historical interpretation or endorsement.