8 Assessing the effectiveness of interruptive EHR alerts via individually randomized trials in the presence of post-randomization contamination

Objectives/Goals: Randomized evaluation of interruptive EHR alerts in improving service delivery or patient health outcomes is usually carried out within the EHR system and in routine hospital operations. However, this study setting leads to increased risk of contamination, where factors outside the original design affect estimation and testing of treatment effect. Methods/Study Population: We focused on the effect of contamination on the design and analysis of individually randomized controlled trials (iRCTs) for the study of the effectiveness of EHR alerts. More specifically, we looked at post-randomization contamination due to clinician-induced correlations. To account for this kind of contamination, we extended the iRCT design to include clinicians’ random effects under the linear mixed effects model (LMEM) framework. We then proceeded to derive sample size and power formulas under different LMEM correlation structures. Using extensive simulation studies, we evaluated the derived formulas and assessed the effect of using a naive two-sample t-test that ignores post-randomization contamination. Results/Anticipated Results: We observed that the level of contamination on sample size, power, and type I error due to clinician-induced correlations was driven by three parameters: variances of the clinicians’ random intercepts and slopes, and the correlation between them. These parameters reflect the way and extent to which clinicians influence participant-level outcomes. For scenarios with common correlation structures from the literature, a test of the treatment effect based on a naive two-sample t-test led to inflated type I error. Using the naive test in the presence of contamination increased the risk of incorrectly concluding that the treatment effect is significant. We did observe valid naive tests with well controlled type I error but under extreme and unusual values of inter-cluster correlations. Discussion/Significance of Impact: Using iRCT as the baseline design for the interruptive EHR alert trials, a naive two-sample t-test may appear reasonable. However, our analytical and simulation findings indicate that a careful consideration of clinicians’ impact on patient-level outcomes in routine hospital operations is necessary for valid hypothesis tests and inference.