Analysis of diagnostic confidence: Application to data from a prospective randomized controlled trial of CT for acute abdominal pain

BACKGROUND: Changes in diagnostic confidence are used as a measure of a test's efficacy. There are several methods for analyzing such data, but it is unclear which are most robust. PURPOSE: To compare analytical methods for assessing diagnostic confidence, applied to data from a prospective study of computed tomography (CT) for acute abdominal pain. MATERIAL AND METHODS: Changes in diagnostic confidence in an illustrative case study were evaluated using five methods: "Basic," "Retained diagnosis," "Omary," "Tsushima," and "Score-based." The case study was a randomized controlled trial of patients admitted to hospital for acute abdominal pain for whom immediate CT was not indicated, comparing "early" CT undertaken within 24 h of admission versus routine standard practice. Admitting surgeons recorded their diagnoses and confidences (5-point scale, 10-90%) both on admission and after 24 h. One- and two-sample t tests, and intention-to-treat and as-treated analyzes, were performed using all five analytic methods. RESULTS: In the case study, 118 patients were randomized to early CT (n=55) or standard practice (n=63). Mean (SD) diagnostic confidence increased between the two time points by 20.7 (25.2)% and 13.0 (24.9)%, respectively. Early CT showed increases in diagnostic confidence by all one-sample analyzes (all P<0.055). Compared with standard practice, early CT showed significant increases in diagnostic confidence on an a) as-treated basis when using Basic, Omary and Score-based analyzes (P<0.045), but not by the other two analytic methods, and b) intention-to-treat, only by Omary analysis. CONCLUSION: The method of analysis used to evaluate diagnostic confidence can influence conclusions about a test's efficacy. Methods incorporating the soundest analytical principles are recommended.