Multiscale adaptive finetuning enhances few shot facial expression recognition

Facial Expression Recognition (FER) is a critical task in the field of computer vision, aiming to recognize the expression classification of an input image. However, existing deep learning-based facial expression recognition methods usually require a large amount of labeled data, which limits their applicability in situations with limited data availability. To address the above problem, we propose a novel multi-scale LoRA adapted fine-tuning facial expression recognition, called MS-LoRA. Specifically, our method is composed of two key stages i.e. pre-training phase and fine-tuning phase. In the pre-training phase, a multi-scale FER large model is designed to extract and learn multi-level general features of facial expressions, which can efficiently and accurately capture the subtle changes of facial expressions and ensure that a wide range of facial features can be recognized by training on datasets. In the fine-tuning stage, the dual-branch FER LoRA module is designed, which can be flexibly embedded into an arbitrary multi-scale model, and is capable of adapting to new facial expression categories with few-shot, thus further improving the model’s adaptability and robustness while preserving the original model’s capability. In addition, we construct five new few-shot datasets with special expressions, including virtual cartoon images, out-of-focus character images, special people images, secondary character images, and pet images. Comprehensive experimental evaluations on established benchmark datasets and the novel few-shot datasets proposed in this study demonstrate the superior performance of our method over leading facial expression recognition (FER) techniques.