What a Systematic Review Cannot See About Itself: Seven measured error classes in search, automated screening, extraction sampling, validation and reporting

Systematic reviews report what they found. They rarely report how their machinery failed while finding it, and the reason is structural: the characteristic errors of automated screening are invisible in a review's own output. A keyword screen that silently deletes eligible studies returns a corpus that looks exactly like a correct one. This paper reports seven error classes found by measuring every stage of one evidence synthesis against something held out from it. The review searched 6,049 records on AI-enabled education interventions in India and comparable low- and middle-income countries, reducing to 3,663 after deduplication, 2,400 after two automated screening stages, and 465 after a full judgement pass over titles and abstracts. Five of the seven only remove records. Screening rules are written as patterns that admit; anything unmatched is excluded by default, so a defect manifests as silence, and silence is indistinguishable from correct exclusion. The largest is string matching standing in for judgement: an audit of 83 exclusions found thirteen records on which a rule had fired wrongly, of which four cost the review a record it would otherwise have carried. Two remove nothing at all, and those are the serious ones. A cell-stratified extraction sample was drawn over a frame later shown to be 81% ineligible, so the inclusion probabilities and intervals computed normally over the wrong population. And the review never had a human reference standard: a sealed 120-record calibration set was built before the screener was written and never coded, so every sensitivity and specificity figure reported for ten weeks was computed against another language model's output. All are withdrawn. The seventh is the one no instrument in the review could have found. A scope decision reversed in the protocol was carried into the screening code and never into the search string, whose exclusion block removed the vocabulary of the studies the reversal existed to admit. Of 975 retrieved teacher-facing records, one contains an excluded phrase. Every validation instrument draws its sample from what the search returned, so none of them could see it. Eight language-model runs judged 2,400 records; the complete human-coded reference standard is twenty records and there is no human second coder. Current reporting guidance asks authors to describe the automated tools they used; it does not ask what those tools were measured against. Data and code: https://doi.org/10.7910/DVN/SCOEFM