Facial Expression Recognition in Anime and Manga Characters: A Comparative Study of Vision Transformers and Convolutional Neural Networks

Facial expression recognition (FER) is a well-established task in computer vision, yet its application to non-photorealistic domains, such as anime and manga, remains largely underexplored. The stylized, exaggerated, and often non-proportional facial features of illustrated characters present unique challenges for deep learning models trained predominantly on realistic imagery. In this work, we construct a balanced dataset of 3000 manga and anime face images spanning six emotion categories (Angry, Embarrassed, Happy, Manic–Euphoric, Sad, Scared) and conduct a systematic comparison of two major deep learning paradigms: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Specifically, we evaluate ResNet-18, ResNet-50, ViT-B/16, and ViT-S/16 under four fine-tuning strategies: linear probing, partial fine-tuning, full fine-tuning, and progressive unfreezing, enabling a controlled comparison of both architectural families and transfer learning depth. Our results show that fine-tuning strategy significantly impacts performance: the best configuration (ViT-B/16 with progressive unfreezing) achieves 81.33% test accuracy (single run, seed 42), compared to 61.33% for the weakest linear probe baseline (ViT-S/16), a gap of 20.00 percentage points. To isolate architectural differences from strategy effects, we note that under full fine-tuning, the only strategy applied identically to all four models, ViT-S/16 (76.00%) outperforms ResNet-18 (74.44%) by 1.56 percentage points and ViT-B/16 (74.22%) by 1.78 percentage points, confirming a modest but consistent architectural advantage for Transformers once backbone adaptation is permitted. Vision Transformers benefit disproportionately from fine-tuning, and the relative ranking of architectures changes across fine-tuning regimes. Confusion matrix analysis reveals persistent cross-class confusion between visually similar emotions (e.g., Happy vs. Embarrassed), while the highly distinctive Manic–Euphoric category is consistently well recognized across all architectures. To the best of our knowledge, this is the first work to conduct a controlled multi-architecture, multi-strategy transfer learning benchmark specifically for FER in anime and manga, revealing findings that are not predictable from photographic FER literature and that carry direct practical implications for model selection in non-photorealistic visual recognition tasks. The anime and manga domain provides a uniquely controlled testbed for studying transfer learning under deliberate stylization, where the domain gap from realistic imagery is not an artifact of image degradation or environmental noise but a principled artistic choice with codified visual conventions; observing that fine-tuning depth dominates architectural choice in this domain suggests the same conclusion likely holds in other non-photorealistic transfer scenarios such as medical illustrations, architectural drawings, and synthetic training data.