Answers are grounded in this paper and cite their sources.
Human consensus is often treated as ground truth in clinical artificial intelligence (AI), yet greater agreement does not necessarily ensure greater clinical relevance. We investigated how expert panel size, composition, and consensus formation influence the reproducibility and clinical relevance of reference standards across three independent pathology cohorts evaluated by international expert pathologists: 50 transbronchial lung cryobiopsy cases from patients with interstitial lung disease assessed by 27 pathologists, 95 resected lung adenocarcinomas evaluated by 11 pathologists, and 596 images of pulmonary granulomatous lesions evaluated by 12 pathologists. Larger panels produced more stable collective judgments but did not improve associations with disease progression or survival in the first two cohorts, instead converging toward consensus classifications with weaker outcome associations. Groups of pathologists identified solely by their diagnostic behavior generated consensus classifications with differing clinical relevance. In lung adenocarcinoma, examination of cases graded differently by the all-rater consensus and a distinct pathologist cluster revealed specific morphological patterns potentially contributing to these differences, suggesting that expert disagreement may contain clinically informative signals. In the granuloma cohort, a modified Delphi process increased agreement, with Krippendorff's α rising from 0.351 to 0.539, while disease-morphology associations changed only modestly. These findings demonstrate that reproducibility and clinical relevance are distinct properties of expert-derived reference standards. Expert disagreement should therefore not be regarded solely as noise to be eliminated, but also as a potential source of clinically meaningful information. Identifying and independently validating outcome-associated morphological features may enable refinement of existing consensus frameworks, supporting the development of more clinically relevant, task-specific ground truth for pathology AI.
No takes yet. Share an insight, caveat, or question.
Kawamoto et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: