Knowledge Distillation for Hybrid CNN–Vision Transformer Architectures: A Systematic Review

Hybrid architectures that combine convolutional neural networks (CNNs) with Vision Transformers (ViTs) have become a dominant design pattern in computer vision, pairing the local inductive bias of convolution with the global relational reasoning of self-attention. This dual advantage comes at a computational cost that frequently excludes hybrid models from edge and mobile deployment, motivating a growing body of work that applies knowledge distillation (KD) either to compress hybrid CNN-ViT models directly or to transfer their heterogeneous knowledge into lightweight students of a different architecture family. Distilling from or into a hybrid model is fundamentally harder than distilling between architecturally homogeneous networks, because it requires reconciling two qualitatively different internal representations: spatially local convolutional feature maps and globally relational attention weights. This review synthesizes more than seventy studies published between 2015 and 2026 that address this problem. The literature is organized along three axes: the direction of cross-architecture knowledge transfer (CNN-to-hybrid, hybrid-to-CNN, hybrid-to-hybrid, and ViT-to-ViT within hybrid families), the type of knowledge transferred (response-based, feature-based, attention-based, and relation-based), and the application domain (general vision, medical imaging, remote sensing, object detection and autonomous driving, agriculture, and action recognition or robotics). A comparative synthesis table of representative methods is provided, and six persistent open challenges are identified: the absence of a unified theory of heterogeneous feature alignment, unexamined hyperparameter sensitivity in multi-component losses, domain-fragmented and incomparable evaluation protocols, scarce validation on physical edge hardware, shallow integration with complementary compression techniques, and underreporting of negative or boundary results. A concrete research agenda is proposed to address these gaps. The review is intended to orient researchers designing distillation pipelines for hybrid vision architectures and to consolidate a literature that currently spans disconnected, domain-specific silos.