Evaluating transportability failures of electronic health record-based risk models for treatment-resistant depression

Earlier identification of risk for treatment-resistant depression (TRD) remains essential, but no risk stratification model has been adopted clinically and none shown to be transportable across sites. To characterize reasons models might fail to transport into new settings, we modeled TRD likelihood using a well-accepted and transparent modeling approach with electronic health records in major depressive disorder at the time of first antidepressant prescription in three large and disparate health systems – Mass General-Brigham, Vanderbilt University Medical Center, and Geisinger Clinic from 2004 to 2022. TRD was defined as provision of electroconvulsive therapy, transcranial magnetic stimulation, vagus nerve stimulation, prescription of ketamine or esketamine or monoamine oxidase inhibitors, or failures of more than two antidepressants. L1-regularized regression applied to sociodemographic features, medications, and ICD-10 diagnostic codes was validated internally and externally with poor performance. C-statistics ranged from 0.51 to 0.65 internally and 0.5 to 0.58 externally. Areas Under the Precision-Recall Curve ranged from 0.07 to 0.15 internally and 0.07 to 0.11 externally with low concordance. We highlight three main failure mechanisms: heterogeneity of TRD as an outcome; biases in source data; site-/practice-level variation. Use of data beyond EHRs and addressing the intrinsic heterogeneity of clinical populations may be necessary to achieve clinically meaningful risk stratification.