Regression Guided by Relative Ranking Using Convolutional Neural Network (R3 CNN) for Facial Beauty Prediction

Facial beauty prediction (FBP) aims to automatically assess facial attractiveness consistently with judgements based on human perception. Most of previous methods formulate FBP as a classification, regression or ranking problem of machine learning. However, humans not only represent facial attractiveness as a score, but also perceive the relative aesthetics of faces. Inspired by this observation, we formulate FBP as a specific regression problem guided by ranking information. Specifically, we propose a general CNN architecture, called R <tex-math notation="LaTeX">³</tex-math> CNN, to integrate the relative ranking of faces in terms of aesthetics to improve performance of FBP. As R <tex-math notation="LaTeX">³</tex-math> CNN consists of both regression and ranking components, it is challenging to train and fine-tune it by existing techniques. To tackle this problem, we propose the following learning schemes for R <tex-math notation="LaTeX">³</tex-math> CNN: 1) a hard pair sampling strategy that generates challenging-to-predicted image pairs and pseudo ranking labels from true rating scores; 2) an assemble loss function that combines regression loss and pairwise ranking loss (PR-Loss), where PR-Loss can be a hinge-form loss or a log-sum-exp pairwise loss; 3) a cascaded fine-tuning method that further improves prediction. Moreover, we build a benchmark dataset, called SCUT-FBP5500, containing 5,500 facial images with diverse properties (male/female, Asian/Caucasian, ages) and labels (face landmarks, rating scores within [1, 5], rating score distribution). Experiments were performed on both the SCUT-FBP and the SCUT-FBP5500 benchmark datasets, where our method achieves state-of-the-art performance on different evaluation settings. Comparisons with related CNN models highlight the effectiveness of the R <tex-math notation="LaTeX">³</tex-math> CNN architecture for FBP.