Answers are grounded in this paper and cite their sources.
Concept-intervention policies are usually compared under a simulated reviewer described by summary statistics, such as error rates per concept or overlap with model errors. We ask whether such summaries determine which of two policies is better. An earlier audit of one 57-patient ultrasound cohort found that opposite rankings of an adaptive policy (P3, entropy times recomputed diagnosis influence) and a fixed-entropy policy (P2) remain attainable after matching every policy's own consumed-answer totals. This paper reports a pre-registered second study. Its confirmatory part replaces synthetic answers with real radiologists' concept ratings from LIDC-IDRI: 622 nodule clusters in 423 patients, seven concepts, four reader fields and 25 cross-fitted concept-bottleneck models (pooled AUROC 0.787). The pre-specified primary result is insufficiency replicates: for every reader field, answer fields with identical consumption-matched summaries rank P3 and P2 either way (attainable ranges 44.4–48.4 pp wide), and no level of a nine-level summary ladder settles the ranking, including the strongest combined set (ranges still at least 20.2 pp wide). Under the real reader fields the two policies are nearly tied (−0.22 to +2.03 pp; every bootstrap interval includes zero). Uniform synthetic noise matched to each reader's per-concept error counts does not reproduce this; it makes P3 look 4.67–5.86 pp worse. Noise placed within model-error signature cells does reproduce it, and real reader errors are concentrated where the models err (pooled odds ratio 3.69). Exploratory extensions in the original cohort reproduce the earlier results exactly, show that finer summaries can settle the ranking there, show that the audit certifies in some settings but not where we predicted, and show that the ambiguity grows rather than shrinks with cohort size.
No takes yet. Share an insight, caveat, or question.
S. M. Abdulla Al Mamun (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: