Sensitivity, net benefit, and informedness of models predicting acute aortic syndrome. (Preprint)

BACKGROUND Acute aortic syndrome is a rare but life-threatening clinical syndrome that can rapidly progress to aortic rupture and death. Symptoms are vague and non-specific, making it challenging to identify. OBJECTIVE We aimed to evaluate prediction models to help clinicians identify acute aortic syndrome based on the data available at the time of presentation. METHODS We combined two existing national datasets of signs and symptoms gathered from patients with and without acute aortic syndrome, from over 30 UK healthcare centres (n = 6,168). Sample incidence was 10.1% (n = 634) against a symptomatic population incidence of 0.26%. We fitted 4,776 prediction models to an 80% ‘training’ split of the data, and then tested on the remaining 20% ‘test’ split. Sensitivity, overall net benefit, and informedness (using Youden’s J) were calculated to represent the perspectives of the clinician, the patient, and the decision modeller. RESULTS The most-common performance was for models to show little to no sensitivity or informedness (< 0.1) and negative overall net benefit. Models with high sensitivity (>0.8) had a range of informedness values, including 0. The only models that had a positive overall net benefit all used the same rule that labelled everyone as having acute aortic syndrome. These “yes to all” models had a sensitivity of 100%, an overall net benefit of only 10%, and informedness value of 0. CONCLUSIONS The perspectives of the clinician, the patient, and the decision modeller need to be considered when developing prediction models for decision support. No model performed well on all evaluation statistics. Difficult trade-offs are revealed, which are exacerbated for rare and severe conditions, such as acute aortic syndrome.