A Hybrid Rule-Based and Machine Learning–Based Clinical Decision Support System to Support Prescription Review: Development and External Validation Study

Background Anticoagulants are high-alert medications with substantial risk of serious bleeding, yet dosing and prescribing errors remain common. Although clinical decision support systems (CDSSs) can mitigate these errors, their impact is constrained by alert fatigue and limited interpretability. Objective We developed and validated a hybrid CDSS that integrates rule-based logic with machine learning to improve the safe use of anticoagulants. Methods This multicenter study used electronic health record data on anticoagulant prescriptions from 3 tertiary hospitals (1 for system development and internal validation and 2 for external validation). The hybrid CDSS combined a knowledge-based rule engine with a machine learning model trained to predict whether an anticoagulant prescription would require pharmacist intervention. The system was iteratively refined through pilot testing, internal validation, and external validation. Results A total of 75,200 anticoagulant prescriptions were used for model development. The final hybrid CDSS comprised 44 patient-specific rules and 1129 drug-drug interaction rules, combined with a CatBoost classifier (version 1.2.5; Yandex) for alert prioritization. During internal validation, 88 (18.9%) alerts were generated; all were technically correct; 88.6% (n=78) were deemed clinically relevant; 86.4% (n=76) were considered clinically useful; and 13.6% (n=12) required pharmacist intervention. In external validation across 2 hospitals, alert rates ranged from 22.6% (7/31) to 32.1% (310/966), with 18.8% (22/117) to 57.1% (4/7) of alerts requiring pharmacist intervention. The hybrid CDSS showed strong discrimination (area under the receiver operating characteristic curve 0.871‐0.963). No false negatives were identified, but the estimates should be interpreted cautiously given the short validation periods and limited number of intervention-requiring prescriptions. Conclusions A hybrid CDSS integrating rule-based logic with machine learning demonstrated high technical accuracy and clinical relevance across multiple institutions, suggesting its potential as a practical tool for supporting pharmacist-led anticoagulant prescription review.