Identification of depressive symptoms: A cause-and-effect based machine learning study

To the Editor: As society continues to advance, individuals are increasingly exposed to various stressors in their personal and professional lives. The World Health Organization claims that nearly 1 billion people worldwide suffer from mental disorders in 2019, including 14% of the world's adolescents.[1] Depression among Chinese adults is a serious public health challenge that requires urgent attention. Due to the combined influence of various life stresses and unhealthy lifestyles, individuals face a significantly increased risk of developing depression.[2] Depression is often misdiagnosed because its symptoms overlap with those of other mental health conditions. A misdiagnosis of depression can have a significant impact on all aspects of an individual's life. In children, a misdiagnosis can lead to inappropriate academic interventions or unnecessary treatments that can adversely affect their school life and overall well-being. For adults, a misdiagnosis can strain relationships, undermine productivity, and lead to long-term emotional distress due to ineffective treatment.[3] Given the challenges in treating depression and its substantial impact on patients' long-term well-being, prevention and early detection are crucial strategies for managing the disease. Compounding the issue, patients with depression, particularly those residing in remote or underdeveloped areas with limited access to healthcare resources, often does not realize they need to be treated. Thus, the development of a practical identification model capable of screening patients for depression holds significant clinical importance. The diagnosis of depression typically involves evaluating a patient's medical history, clinical symptoms, and duration of illness, as well as considering physiological signs such as the results of electroencephalography, skin conductance tests, and eye movement analysis. The main internationally recognized diagnostic criteria for depression are the 10th revision of the International Classification of Diseases (ICD-10) and the Diagnostic and Statistical Manual of Mental Disorders, fourth edition (DSM-IV).[4] However, the clinical application of these criteria usually requires evaluation with specialized instruments and by trained physicians, which can make patient visits more complicated. Researchers are therefore striving to develop low-threshold intelligent detection methods that would streamline the diagnostic process. The purpose of this study was to develop a depression prediction model based on causal inference and machine learning. In this study, we applied a causal inference algorithm to select characteristic variables associated with depression from patients' clinical information and Symptom Checklist 90 (SCL-90) data; the selected characteristic variables were then inputted into a machine learning model for the prediction of depression. We hoped to create a low-cost prediction model to address the current lack of depression-screening tools, particularly in underdeveloped areas. This study was divided into four steps. In the first step, the data were preprocessed. In the second step, a complete causal network structure map was constructed using the collected clinical characteristics and information about the 9 scale dimensions of the SCL-90 to find factors that have a direct causal relationship with a diagnosis of depression, i.e., the main factors. In the third step, the main factors that affect the diagnosis of depression, as identified through the causal network structure map, were inputted as variables into the machine learning model for predicting depression. In the fourth step, the performance of the model was evaluated. We collected data from the actual visit records of 4763 patients with depression who were treated in the psychiatric outpatient clinic of the Seventh People's Hospital in Hangzhou, China, and 2998 healthy controls between June 2019 and July 2021. In addition, we gathered data from 2998 healthy controls. The SCL-90 test was completed in an independent setting under the guidance of experienced psychometric staff, and a computerized system was used to tally the patients' scores on the 9 scale dimensions of the SCL-90. Considering the possible predictors of depression and our own clinical experience, we collected the following clinical information from each participant: (1) subjective complaints, i.e., the patient's subjective feelings and (2) demographic variables such as age, gender, education, and marital status. The data were independently confirmed by two additional researchers. With the patients' chief complaints, clinical characteristics (e.g., age, gender, marital status, and education), and the 9 scale dimensions from the SCL-90, we developed a feature selection model based on a causal inference algorithm to filter out the clinical characteristics and scale dimensions that were directly and causally related to depression. We applied the maximum relevance–minimum redundancy (mRMR) algorithm to sequentially find the set of causal nodes, which reduced the complexity of the next step in which we applied the conditional independence test. A causal skeleton was constructed such that for any variable dataset X = {x1, x2, …, xn}, the variables that are directly and causally related to the target node y could be found. First, the node xm with the largest dependency on node y was incrementally searched using the mRMR search algorithm to form the candidate factor set S(x), and then, the conditional independence test method was used to remove the candidate nodes and the non-parent-child nodes of S(x). These steps were repeated in S(x) to remove nodes that had no direct causal relationship with the target node y. The feature selection process resulted in the selection of the following clinical features: age, gender, marital status, negativity, sleep problems, nervousness, upset, reduced speech, and suspicion. In addition, the following scale dimension features were selected: depression, phobic anxiety, and obsessive-compulsive. Using the Python Scikit-learn package (https://scikit-learn.org/stable/), we applied four popular machine-learning models to the selected features: K-nearest neighbor (KNN), multilayer perceptron (MLP), support vector machine (SVM), and Random Forest (RF). To improve the generalizability of the predictive models, we adopted a 10-fold cross-validation method during the training process for internal validation. Specifically, we divided the dataset into an average of 10 parts, ensuring that the proportion of observations of different categories in each part was as consistent as possible with the whole dataset. Each part was used as a test set in turn, and the remaining part was used as a training set to train the model; the effect of training was verified on the test set. All the data from January 2019 to June 2021, were used as the training set, and the data from July 2021 to July 2021 were used as the test set to ensure that the population for validation and the modeling population were distinct, thereby ensuring the validity of the validation. To assess the predictive performance of the four prediction models in detecting depression, we calculated various evaluation metrics, including the F1_score, accuracy, sensitivity, precision, and the area under the curve (AUC). First, we compared the performance of the machine learning models during the 10-fold cross-validation. The results of training set were as follows: (1) KNN (accuracy = 85.3%, sensitivity = 82.7%, precision = 87.5%, F1_score = 90.0%, AUC = 0.88); (2) MLP (accuracy = 76.0%, sensitivity = 85.8%, precision = 88.7%, F1_score = 82.1%, AUC = 0.81); (3) SVM (accuracy = 84.1%, sensitivity = 87.4%, precision = 88.7%, F1_score = 86.0%, AUC = 0.86); and (4) RF (accuracy = 88.3%, sensitivity = 92.3%, precision = 90.2%, F1_score = 91.2%, AUC = 0.90). The above indices were recalculated in the validation tests: (1) KNN (accuracy = 84.2%, sensitivity = 83.4%, precision = 78.0%, F1_score = 80.5%, AUC = 0.87); (2) MLP (accuracy = 75.9%, sensitivity = 87.2%, precision = 81.3%, F1_score = 82.1%, AUC = 0.80); (3) SVM (accuracy = 83.9%, sensitivity = 84.1%, precision = 88.4%, F1_score = 84.1%, AUC = 0.85); and (4) RF (accuracy = 88.0%, sensitivity = 89.1%, precision = 90.5%, F1_score = 87.8%, AUC = 0.90). To evaluate the performance of the depression-prediction models in a clinical setting with a relatively low-disease prevalence, we tested the models in four clinical datasets with decreasing proportions of patients with depression. In total, 1000 patients with depression and 1000 healthy controls were randomly selected from an external dataset gathered from Hangzhou Seventh People's Hospital between July 2021 and December 2021 and included in this clinical test. In the four datasets, the ratios of patients to healthy controls were approximately 1:1, 1:2, 1:3, and 1:10. To reduce the sampling error, we performed four sampling analyses with replacement in each dataset, except for the dataset with a 1:1 ratio of patients to controls. When the ratio of patients vs. controls was 1000:1000 (1:1), we obtained the following results: (1) KNN (accuracy = 83.3%, sensitivity = 82.7%, precision = 87.5%, F1_score = 87.0%, AUC = 0.89); (2) MLP (accuracy = 76.0%, sensitivity = 85.6%, precision = 84.7%, F1_score = 80.1%, AUC = 0.81); (3) SVM (accuracy = 83.1%, sensitivity = 84.4%, precision = 88.7%, F1_score = 86.0%, AUC = 0.87); and (4) RF (accuracy = 85.3%, sensitivity = 88.3%, precision = 86.2%, F1_score = 91.2%, AUC = 0.91). The results for the sample with a 500:1000 (1:2) ratio were as follows: (1) KNN (accuracy = 84.2%, sensitivity = 83.4%, precision = 78.0%, F1_score = 87.5%, AUC = 0.88); (2) MLP (accuracy = 75.9%, sensitivity = 87.2%, precision = 85.3%, F1_score = 81.1%, AUC = 0.80); (3) SVM (accuracy = 83.9%, sensitivity = 84.5%, precision = 88.8%, F1_score = 84.1%, AUC = 0.85); and (4) RF (accuracy = 86.0%, sensitivity = 89.1%, precision = 86.5%, F1_score = 91.8%, AUC = 0.90). The sample with a 333:999 (1:3) ratio yielded the following results: (1) KNN (accuracy = 85.0%, sensitivity = 83.7%, precision = 87.5%, F1_score = 90.0%, AUC = 0.87); (2) MLP (accuracy = 76.1%, sensitivity = 85.7%, precision = 85.7%, F1_score = 82.1%, AUC = 0.80); (3) SVM (accuracy = 84.1%, sensitivity = 84.4%, precision = 88.8%, F1_score = 87.0%, AUC = 0.84); and (4) RF (accuracy = 88.3%, sensitivity = 89.3%, precision = 90.2%, F1_score = 91.2%, AUC = 0.89). Finally, the results for the 100:1000 (1:10) sample were as follows: (1) KNN (accuracy = 85.2%, sensitivity = 84.4%, precision = 88.0%, F1_score = 90.5%, AUC = 0.85); (2) MLP (accuracy = 76.1%, sensitivity = 87.3%, precision = 86.5%, F1_score = 82.3%, AUC = 0.80); (3) SVM (accuracy = 84.9%, sensitivity = 84.4%, precision = 88.9%, F1_score = 87.1%, AUC = 0.83); and (4) RF (accuracy = 88.4%, sensitivity = 89.4%, precision = 90.6%, F1_score = 92.8%, AUC = 0.87). The depression-prediction models showed good discrimination between patients with depression and healthy individuals, with high accuracy and AUC values. In addition, the depression-prediction models achieved stable performance in the four subgroups in the clinical setting, maintaining acceptable accuracy and AUC values even in the 1:10 dataset. In summary, in this study, we identified 12 predictive factors (age, gender, marital status, negativity, sleep problems, nervousness, upset, reduced speech, suspicion, depression, phobic anxiety, and obsessive-compulsive) that have a causal relationship with depression. Unlike in previous research,[5] we introduced causal inference algorithms in the feature-selection process, which enhanced the interpretability of the selected features. We have developed an explanatory and practical identification model that effectively detects depression using only these readily available predictive factors. The model's high feasibility and almost negligible cost make it a promising solution for depression-screening, particularly in underdeveloped regions. This research provides a valuable approach to address the issue of depression screening and offers insights for further studies in related domains. Ethics statement This study involving human participants was reviewed and approved by the Ethics Committee of Hangzhou Seventh People's Hospital (No. 2023-009). The informed consent of patients was waived by the institutional Ethics Committee due to the retrospective design. All healthy participants provided written informed consent. Funding The Project of Natural Science Foundation of Hunan Province (No. CX20221185) and the Science Foundation of Hunan Province (Nos. 2022JJ70072 and 2021JJ70151) supported this work.