Multimodal AI for stroke identification during medical emergency calls

Background/Introduction Stroke is a well-known cause of death and disability. Rapid, accurate dispatcher recognition during emergency medical calls is critical for timely reperfusion, yet call takers often work with fragmented and noisy information. We hypothesised that integrating call transcripts with structured patient data could improve stroke triage in the Emergency Medical Communication Centre (EMCC). Purpose To evaluate whether a multimodal AI pipeline developed within the Artificial Intelligence Support in Medical Emergency Calls (AISMEC) project can improve stroke triage during emergency medical calls by integrating call transcripts and structured health data, and to compare its performance with EMCC operator assessments and transcript-only models. Methods We performed a retrospective, machine learning development and validation study using audio logs from 2022 from Bergen EMCC (population ≈470,000). Two corpora were used: (i) 473 audio files with manual transcripts to finetune an automatic speech recognition (ASR) model (nb-whisper-large), and (ii) 2,627 labelled calls (329 stroke) to train and evaluate a ModernBERT-large classifier. Structured data from electronic health records included vascular risk factors and disease, laboratory measurements, and prescribed medications including primary and secondary prevention. Fivefold cross validation with class weighted loss mitigated imbalance. Model performance was primarily evaluated using discrimination metrics, including area under the receiver operating characteristic curve (ROC), recall (sensitivity), and precision (positive predictive value), and F1-score. Both macro-averaged and weighted-averaged F1-scores were reported. EMCC operator baseline was derived from review of EMCC documentation on the held-out test set. Results ASR finetuned for medical emergency calls, reduced word error rate from 8.3 to 7.2. The multimodal classifier achieved a stroke precision=0.80, recall=0.76, F1=0.78; non-stroke precision=0.97, recall=0.97, F1=0.97; weighted F1=0.95. ROC-AUC was 0.856. Compared with the human baseline (stroke F1=0.73), overall accuracy improved by 1.3 percentage points. In a simulated live call analysis, performance rose steeply within 35–55 seconds (~300–600 tokens) and then plateaued. Multimodal input consistently outperformed transcripts only. Conclusion(s) A multimodal AI pipeline combining emergency call transcripts with structured patient data improved stroke triage beyond EMCC operator performance and transcript only models, with reliable predictions available within the first minute of the call. Prior to clinical implementation aimed at reducing prehospital delay in the stroke chain of survival, external validation in larger, more diverse datasets and workflow evaluation is warrantedGraphical abstract