1021: A Novel Framework for Enhanced Clinical Decision Making and Alert Triage Using Vision-Language Model

Introduction: Current applications of large language models (LLMs) for triage mainly focus on processing unstructured text and structured tabular data from electronic health record (EHR). In contrast, we present a novel framework that leverages a Vision-Language Model (VLM) to provide immediate triage and acute care assessments by interpreting a single image of vital sign trends following a critical alert. Methods: Our iterative development included studies comparing multiple input formats including tabular data, various images, and a multi-modal combination of both. From this, we identified the single two-hour vital sign trend image as the optimal input for the VLM before deploying the framework to an ICU dashboard. During a critical alert, the deidentified image of physiological vital signs (SpO2, respiratory rate, heart rate and blood pressure over 2-hours) is passed to a VLM (Gemini 2.5). The model is guided by a structured, modular prompt that provides summary detailing data quality issues like missingness: a Clinical Interpretation based on alert thresholds; a Triage Recommendation; and an Acute Care Need Assessment in a predefined format. Results: Our single-image VLM approach achieves a 50-second processing time by avoiding the high-latency EHR data retrieval required by traditional text-based Retrieval-Augmented Generation (RAG) frameworks. The structured prompting strategy enhances safety by constraining the model’s output and mitigating hallucination risk. The framework also demonstrated high output fidelity, with a 99% schema adherence rate, ensuring output is consistently machine-readable for dashboard integration. This single-image approach is highly efficient, reducing token consumption by an estimated 80% compared to text-based EHR retrieval while eliminating the significant technical overhead of complex Application Programming Interface integrations. Conclusions: The VLM approach demonstrates significant architectural and efficiency advantages over traditional text-based RAG systems. We plan to evaluate if these benefits translate to real-world impact in a prospective clinical trial focused on improving clinician workflow. The primary goal of this trial will be to assess the framework’s ability to mitigate alert fatigue, a challenge driven by frequent false positive alarms in the ICU.