What question did this study set out to answer?

To compare the performance of two FDA-cleared AI solutions in detecting vessel occlusions on CT head angiograms.

February 2, 2026

Abstract WP270: Comparative Evaluation of Two Commercial AI Solutions for Vessel Occlusion Detection in a Real-World Clinical Setting

Key Points

To compare the performance of two FDA-cleared AI solutions in detecting vessel occlusions on CT head angiograms.
Retrospective analysis of 1,557 CT angiogram cases processed by two AI vendors over 18 weeks.
Utilized a natural language processing algorithm to classify radiology reports.
Concordance between vendors' outputs was assessed, with discordant cases reviewed for ground truth determination.
Calculated sensitivity, specificity, and predictive values for both vendors.
Vendor A showed 92.4% sensitivity and 97.6% specificity, while Vendor B showed 35.4% sensitivity and 98.1% specificity.
Vendor A detected 92.6% of large vessel occlusions and 92.3% of medium vessel occlusions, whereas Vendor B detected 70.4% of LVOs and only 17.3% of MeVOs.
The findings highlight notable performance differences in medium vessel occlusion detection between the two AI solutions.

Abstract

Purpose: To compare the performance of two FDA-cleared artificial intelligence (AI) solutions, Vendor A (Aidoc, Tel Aviv, Israel) and Vendor B (Viz.ai, San Francisco, USA) in detecting vessel occlusions (VOs) on CT head angiograms within a large academic healthcare environment. Materials and Methods: This retrospective study included cases processed by both Vendor A and Vendor B over an 18-week period (June 1–September 27, 2024). A natural language processing (NLP) algorithm classified radiology reports. When NLP, vendor A, and vendor B outputs were concordant, this was assumed as ground truth; cases with any discordance underwent review and adjudication to establish ground truth. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Results: A total of 1,557 consecutive CT angiogram cases were reviewed. NLP algorithm identified 113 true positive reports, corresponding to a prevalence of 7.3% (113/1,557). With 402 cases exhibiting discordance. Vendor A demonstrated a sensitivity of 92.4%, specificity of 97.6%, PPV of 76.8%, and NPV of 99.3%. Vendor B demonstrated a sensitivity of 35.4%, specificity of 98.1%, PPV of 62.2%, and NPV of 94.5% as seen in Table 1. Vendor A detected 92.6% of Large Vessel Occlusions (LVO) and 92.3% of Medium Vessel Occlusions (MeVO). Vendor B detected 70.4% of LVOs and only 17.3% of MeVOs, highlighting notable performance differences in MeVO detection as seen in Table 2. Conclusion: In this real-world clinical comparison, Vendor A demonstrated higher sensitivity and stronger agreement with ground truth compared to Vendor B in detection of VO. Vendor A detected 92.6% of LVOs compared to 70% for Vendor B, with the overall performance gap attributable to Vendor A’s greater accuracy in identifying MeVOs.These findings provide valuable comparative data and underscore the importance of evaluating both diagnostic accuracy and statistical agreement when selecting AI solutions for stroke triage workflows. Clinical Relevance Statement: AI algorithms designed for VO detection may vary significantly in diagnostic performance and clinical integration. This study emphasizes the importance of critical vendor assessment, particularly regarding detection sensitivity and agreement with expert adjudication, to guide optimal AI adoption in acute stroke care environments.

Read Full Paperexternally

AIに質問

Bookmark

View Full Paper