Bone Metastasis Detection at CT with Deep Learning Models Trained Using Multicenter, Multimodal Reference Standards: Development and Evaluation

Purpose To develop and validate deep learning models for detecting bone metastases on abdominal and thoracic CT scans, considering lesion visibility, and to compare model performance against human readers. Materials and Methods This retrospective multicenter study included CT scans from patients with bone metastases at four medical centers (August 2013-October 2021). MRI and PET-CT served as reference standards to categorize lesions as visible, indeterminate, or invisible based on CT visibility. Two nnU-Net models were trained: Model 1 with only CT-visible metastases and Model 2 with both visible and indeterminate metastases. Lesion-level performance was evaluated using precision and recall. Scan-level performance was evaluated using area under the receiver operating characteristic curve (AUC). Model performance was compared with that of three musculoskeletal radiologists and three radiologists in training. Results 502 CT scans from 332 patients (mean age, 64.2 years ± 11.2 [SD]; 171 males) with 4,999 bone metastases were included. While lesion-level precision was similar between both models (Model 2: 80.1%; Model 1: 78.8%; P = .41), Model 2 achieved higher recall (41.8% vs 33.9%; P < .001) and among visible lesions (53.6% vs 44.7%; P < .001). Both models’ precision exceeded that of radiologists in training (66.6%; P < .003) and musculoskeletal radiologists (66.5%; P < .004). Only Model 2 achieved recall comparable with that of both radiologists in training (39.4%; P = .37) and musculoskeletal radiologists (43.8%; P = .47), as well as a comparable scan-level AUC (0.80 [95% CI: 0.67, 0.90]; P > .05). Conclusion The deep learning model trained with multimodal reference standards achieved expert-level bone metastasis detection performance on body CT.