A smartphone AI application for cardiac device identification from chest radiographs misidentifies one in three devices: an independent dual-rater validation

Background Accurate identification of cardiac implantable electronic devices from chest radiographs is critical for device programming, MRI safety, and emergency care. Smartphone artificial intelligence applications offer rapid point-of-care identification, yet clinicians adopt these tools without robust evidence of accuracy. No study has independently validated such an application against a ground-truth labelled dataset. Purpose We assessed the diagnostic accuracy, confidence calibration, and inter-rater reproducibility of a smartphone artificial intelligence application for cardiac device manufacturer identification. We hypothesised that accuracy would be insufficient for standalone clinical use and would vary by device type. Methods Two blinded raters independently tested 384 chest radiographs from the PhysioNet cardiac device dataset. The dataset included 100 images per manufacturer (Biotronik, Boston Scientific, Medtronic, St. Jude Medical) across five device types: defibrillators (n=118), implantable loop recorders (n=85), pacemakers (n=75), resynchronisation defibrillators (n=64), and resynchronisation pacemakers (n=42). For each image, we recorded the top prediction, confidence score, and second prediction. We calculated accuracy with 95% Wilson confidence intervals and inter-rater agreement with Cohen’s kappa. Results Overall accuracy was 52.6% (Rater 1) and 49.2% (Rater 2). Even considering the top two predictions, accuracy reached only 62.8%/63.2%. Within the intended scope (pacemakers and defibrillators, n=299), one in three devices was still misidentified (65.9% and 60.9%). Performance varied by device type (p<0.001): resynchronisation defibrillators achieved 90.6%/81.2%, but standard defibrillators only 58.5%/55.1% and pacemakers 54.7%/48.0% (Figure). Boston Scientific had the highest sensitivity (81.0%/66.0%); Biotronik (35.0%/40.0%) and St. Jude Medical (44.0%/38.0%) were frequently misclassified. Implantable loop recorders, outside the intended scope, achieved only 5.9%/8.2%. The application returned high-confidence “Unknown” results for 14-20% of images. Confidence calibration was poor: 29-39% of maximum-confidence predictions were incorrect. Inter-rater agreement was only moderate (kappa=0.530). Conclusion This first independent dual-rater validation demonstrates that a smartphone artificial intelligence application for cardiac device identification is not sufficiently accurate for standalone clinical use. One in three devices was misidentified within the intended scope, confidence scores were poorly calibrated, and results varied between raters. While resynchronisation defibrillators achieved accuracy above 80%, performance for standard defibrillators and pacemakers was inadequate for safe clinical decision-making. Improved training data, confidence calibration, and standardised image acquisition are needed before safe bedside deployment.