Enhancing personalized suicide risk prediction for VA patients by integrating discrete natural language processing models

To improve the identification of Veterans at risk for suicide, the U.S. Department of Veterans Affairs (VA) developed REACH-VET, a suicide risk classification metric. Our previous work demonstrated that incorporating natural language processing (NLP) and developing targeted models for distinct suicide risk-tiers could enhance REACH-VET’s predictive accuracy. This study evaluates the benefits of two NLP methods and compares their predictive performance across risk-tiers. We created a sample of VA patients who either died by suicide in 2017–2018 (cases) or remained alive during that period (controls), stratified by suicide risk (high, moderate, low). We analyzed unstructured electronic health record (EHR) notes using two NLP models: 1) theory-based, closed-vocabulary “semantic” methods, and 2) data-driven, open-vocabulary “count” methods. We then developed eXtreme Gradient Boosting (XGBoost) classification models using semantic, count, and hybrid (count and semantic) variables and calculated area under the receiver operating characteristic curve to assess predictive accuracy. Generally, classification models using semantic variables performed worse than count or hybrid variables. The highest added benefit was seen for low- and moderate-risk patients who achieved incremental improvements in performance over and above leading predictive benchmarks. By using different NLP techniques on unstructured EHR data, our approach improves predictive accuracy for lower-risk patients, expanding suicide prediction for patient populations who are poorly understood and frequently underserved.