A Known-Answer Audit for Deployed Measurement Systems: Procedure and a Case Study

Systems that rank online communities by "health" are deployed without any way to check whether the rankingmeasures what it is used for. The index emits plausible numbers, the dashboard orders communities by them, andnothing in the pipeline can detect that the ordering is wrong, because no ground truth exists against which"wrong" could be defined. This is a construct-validity failure of the kind Jacobs and Wallach [1] describe asendemic in computational measurement, and it is invisible to the testing practices that production machinelearning systems normally apply [2], [3].We describe a lightweight audit procedure for such systems and report what it found when applied to onedeployed community-health pipeline. The procedure constructs a benchmark in which the answer is recorded inadvance and withheld from every pipeline stage, then subjects each component of the index, and the composite, tothe same known-answer test. A planted signal can only test whether an instrument detects what it was built todetect; we therefore treat the benchmark as a test instrument, never as evidence about online communities.Applied to a five-layer pipeline serving a live dashboard, the audit surfaced four defects invisible from thesystem's outputs. (1) Two of the health index's four components are constant for 42 of 60 scored communities, so45% of the index's weight is inert for 70% of its population. (2) A third component separates the classes at chance(AUC 0.498). (3) The four-component composite does not measurably outperform its single strongest input(paired ΔAUC, toxicity minus composite, +0.075; 95% CI [−0.027, +0.203]). (4) The pipeline's Temporal FusionTransformer [4] recovers the planted trajectory at slope AUC 0.732 (95% CI [0.601, 0.849]), while an ordinaryleast-squares slope over the same daily series recovers it at 1.000 (paired ΔAUC +0.268, 95% CI [+0.151,+0.399]): the deep model underperforms a one-line baseline on the task it was adopted to perform, echoingfindings in recommendation [5] and forecasting [6].On a real corpus of 116,853 comments across 24 communities, ranking by the present-state index and by a trendaware score agree only weakly (Spearman ρ = 0.383, p = 0.065; 13 of 24 communities move five or more places),and the two top-five lists share only two members. The instrument choice materially changes the moderationqueue.None of these defects was visible before the audit and all four are actionable. The first is detectable by countingdistinct values alone, and the procedure's checklist gives label-free checks for the rest. We give the procedure as aseven-item checklist, release the code and data, and argue that measurement audits of this kind are cheap enoughto be routine and are currently almost never run.