Exploring Predictors of Facial Impressions: An Integrated Approach Utilizing VAE, Grad-CAM, and GPT-4V

In recent years, the relationship between human facial features and the impressions they produce has been modelled computationally using geometric morphometrics and deep learning methods. However, appropriately labeled face image datasets often contain too few samples to train deep learning models. Additionally, predictor analyses of these models remains subjective. We combined geometric morphometrics with a variational autoencoder (VAE) to generate facial images corresponding to impressions of attractiveness, dominance, and sexual dimorphism. The resulting models can contribute to the understanding of critical facial features that are difficult to verify using psychological experiments alone. Furthermore, we combined gradient-weighted class activation mapping (Grad-CAM) and a large language model (GPT-4V) to explain the model predictors in words. The proposed approach combining a large language model with explainable artificial intelligence (VAE for the input values; Grad-CAM) has affective engineering applications in facial editing, beautification, and makeup recommendations.