A Hybrid BERT-CNN Approach for Depression Detection on Social Media Using Multimodal Data

Abstract Due to the absence of early facilities, a large population is dealing with stress, anxiety, and depression issues, which may have disastrous consequences, including suicide. Past studies revealed a direct relationship between the high engagement with social media and the increasing depression rate. This research initially creates a dataset with text, emoticons and image data, and then preprocessing is performed using diverse techniques. The proposed model in the research consists of three parts: first is textual bidirectional encoder representations from transformers (BERT), which is trained on only text data and also emoticons are converted into a textual form for easy processing; second is convolutional neural network (CNN), which is trained only on image data; and the third is the combination of best-performing models, i.e. hybrid of BERT and CNN (BERT-CNN), to work on both the text and images with enhanced accuracy. The results show the best accuracy with BERT, i.e. 97% for text data; for image data, CNN has attained the highest accuracy of 89%. Finally, the hybrid approach is compared with other combinations and previous studies; it achieved the best accuracy of 99% in the categorization of users into depressive and non-depressive based on multimodal data.