Attention Pooling-Based IndoBERT with High-Confidence Pseudo-Labeling for Indonesian Mental Health Classification

Mental health disorders are a significant global concern, while social media provides user-generated text reflecting emotional and psychological conditions. However, Indonesian mental health classification remains challenging due to limited high-quality annotated datasets and limitations of conventional document representation methods. This study develops an Attention Pooling-Based IndoBERT framework integrated with high-confidence pseudo-labeling for multiclass Indonesian mental health classification. Unlabeled YouTube comments were automatically labeled using a BERT teacher model evaluated on an independent test set and filtered using a confidence threshold of 0.90, retaining 1,213 high-confidence samples (22.20%). The Attention Pooling mechanism assigns adaptive weights to contextual token representations, while Focal Loss, label smoothing, and learning-rate scheduling address class imbalance. The proposed model achieved 97.80% Accuracy and 79.63% Macro F1-score on an independent test set of 182 samples. Ablation analysis showed higher test Accuracy for Attention Pooling than CLS, Mean, and Max Pooling. McNemar’s test indicated significant differences between Attention Pooling and RoBERTa, standard IndoBERT, CLS, Mean, and Max Pooling, whereas the difference from BERT was not significant. These findings support Attention Pooling as an effective representation strategy for Indonesian mental health classification in low-resource settings.