Validation study reveals severe generalization gaps in deep-learning tuberculosis classifiers across external cohorts, highlighting that standard segmentation mitigations fail to prevent model...
Deep networks achieve near-perfect tuberculosis (TB) detection on chest radiographs within a single dataset, but such figures are rarely validated across institutions. We train a MobileNetV2 classifier on a widely used public TB radiography database and evaluate it, under a leakage-safe protocol with bootstrap confidence intervals, on two independent National Library of Medicine cohorts (Montgomery and Shenzhen). In-domain area under the ROC curve (AUC) reaches 0.9995 but falls to 0.70-0.78 externally -- an approximately 0.25 AUC generalisation gap that five-fold cross-validation confirms is stable (0.25 +/- 0.02). We decompose the accompanying collapse of sensitivity to near zero into a threshold-transfer component (the optimal operating point shifts roughly fivefold between sources) and a discrimination ceiling of about 70% sensitivity at 70% specificity, and a duplicate-image audit establishes that the external cohorts share no overlap with the training data. We then test the two most commonly recommended mitigations. Intensity harmonisation with augmentation and multi-source training improves external AUC only marginally (+0.02). Lung segmentation, using a U-Net segmenter of high quality (Dice 0.975) producing anatomically sound masks, fails to close the gap and degrades unbiased external AUC in every cross-validation fold (paired decrease 0.057 +/- 0.011), contrary to prior reports for COVID-19 radiography. The failure is moreover architecture-dependent: identically trained DenseNet-121 and EfficientNet-B0 backbones match MobileNetV2 in-domain (all >= 0.999) yet each collapses on a different external cohort (DenseNet to chance on Shenzhen, 0.503 +/- 0.025; EfficientNet to near-chance on Montgomery, 0.541 +/- 0.061), so the two cohorts rank the three architectures in entirely different orders. A frozen-feature probe shows these failure profiles already exist in the ImageNet-pretrained representations before any fine-tuning, and masking homogenises all three backbones into a common 0.61-0.77 band well below in-domain performance. We conclude that the TB-CXR generalisation gap is real, large, and resistant to the standard playbook; that single-cohort external validation can invert architecture rankings; and that out-of-source validation on multiple cohorts is a prerequisite for trusting such models.
No takes yet. Share an insight, caveat, or question.
Alvin Chow Chow (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: