Improving patient triage classification in the emergency department using natural language processing and machine learning algorithms: a retrospective cohort study

Triaging in the Emergency Department (ED) is the initial assessment determining a patient’s acuity level. Precise patient stratification at this stage is crucial as it can reduce patient safety risks, determine optimal healthcare treatments, and optimize costs. Traditional triage systems have been associated with under-triage and over-triage problems, as well as high variability and individual bias. This article proposes a classification framework based on machine learning (ML) and natural language processing (NLP) that leverages both structured and unstructured data to enhance the triage classification of Emergency Severity Index (ESI) patients. We propose a framework for classifying patient triage across the three levels of the ESI system using three data configurations: structured data only, unstructured data only, and a combination of both, along with strategies for handling imbalanced data. We tested four classification algorithms: eXtreme Gradient Boosting (XGBoost), Random Forest (RF), Multilayer Perceptron Neural Network (MLP), Logistic Regression (LR) and an NLP method: Term Frequency—Inverse Document Frequency (TF-IDF), both without and with two oversampling techniques: Synthetic Minority Over-sampling Technique (SMOTE) and ADAptive SYNthetic (ADASYN). A feature selection step was performed using variable importance plot (VIP) for all analysis sets. To optimize the hyperparameters of the four classification algorithms, we used grid search on the training set and evaluated their performance on the validation set. A 10-fold cross-validation was performed on the training and validation sets and eight performance metrics were calculated. The methods were tested using a dataset from an ED of a large private hospital in Brazil. The study included 64,382 adult patients, of whom 38.3% were triaged at levels 1 and 2. The model with the highest ROC-AUC was XGBoost, achieving 0.871 on the validation set and 0.873 on the test set using structured plus unstructured data. Algorithms using the complete datasets generally outperformed those using only structured data. Overall, models that incorporated NLP achieved the highest ROC-AUC. The NLP model used to vectorize chief patients’ complaints enhanced performance metrics by 4% to 11% on both validation and test sets. I Integrating patients’ chief complaints through NLP enhanced the performance of ML algorithms for patient triage in the ESI system in Brazil. The inclusion of unstructured data provided by NLP resulted in more accurate and reliable triage predictions.