Cross-modal consistency gated fusion network for selfie-based body satisfaction prediction

Accurately predicting users’ satisfaction with their body image in selfies is of significant importance for understanding body perception, preventing psychological issues, and developing personalized health services. To this end, this study proposes an innovative selfie-based body satisfaction prediction model that integrates Bidirectional Encoder Representations from Transformers-Convolutional Neural Network (BERT-CNN). The model aims to achieve more precise inference of psychological states by deeply integrating multimodal information. It employs a dual-branch architecture: the text branch leverages a pre-trained BERT model to extract deep semantic features, capturing sentiment tendencies and self-evaluation in descriptions, while the image branch uses Residual Network-50 (ResNet-50) combined with a channel attention mechanism to focus on key visual cues such as facial expressions and body posture in selfies. A core innovation lies in the cross-modal attention fusion module, which enables fine-grained interaction between text and image features through a bidirectional attention mechanism. An adaptive gating unit dynamically integrates information from both modalities, producing a more discriminative joint representation. Experiments conducted on the Selfit dataset show that the model achieves a Mean Absolute Error (MAE) of 0.310 and a Mean Squared Error (MSE) of 0.211 on the test set. When converting predicted values into discrete satisfaction levels for classification tasks, the model attains an accuracy of 83.0% and a weighted F1 score of 0.828, significantly outperforming single-modality approaches and existing state-of-the-art multimodal fusion baselines. This study validates the effectiveness of deep text-image integration for predicting body satisfaction and also offers a methodological reference for multimodal psychological computing. The approach has potential applications in digital mental health assessment and intelligent content understanding.