An analytics framework for timing-aware evaluation of prediction-triggered clinical alerts

Clinical alert systems are commonly evaluated using predictive performance and decision-analytic measures that treat all true-positive alerts within a prediction horizon as beneficial. For time-sensitive decisions, this assumption may be too permissive because a correct alert may arrive too late to support action. This study introduces Actionable Net Benefit (ANB), a timing-aware extension of net benefit that counts a true-positive alert as beneficial only when it satisfies a prespecified minimum lead-time requirement. ANB was evaluated using a retrospective inpatient falls case study comprising 58,803 discharge-aware patient-day alert opportunities from 5451 encounters, including 96 positive 24-hour windows. A ridge-penalized logistic model was selected through encounter-stratified cross-validation and evaluated using nested encounter-level out-of-fold predictions. The model achieved an out-of-fold AUROC of 0.810 and AUPRC of 0.00715, compared with 0.560 and 0.00398 for the Hester-Davis comparator. Operational capacity was applied within each 7:00 AM alert cycle and was separated from the threshold probability used to weight false-positive burden. Under the reference analytic scenario of 5% per-cycle capacity, the simulated policy generated 2336 alerts and selected 19 positive windows, of which 13 remained actionable under the reference 6-hour lead-time requirement. At p t = 0.005 , ANB decreased from 1.502 per 10,000 alert opportunities at L = 0 to 0.277 at L = 6 . The corresponding encounter-clustered bootstrap 95% intervals were 0.081 to 3.850 and -0.979 to 2.084, and the paired change was -1.225 (-2.592 to -0.409). At L = 6 , ANB ranged from 1.469 to -2.124 across the lower and higher threshold-probability scenarios. Schedule-specific recalibration improved probability alignment for 12-hour and 6-hour horizons, while the static daily predictor design produced lower ANB under more frequent schedules. ANB provides a structured approach for evaluating whether candidate alert policies remain useful after operational capacity, preference weighting, and response-time requirements are considered.