Lightweight Image Restoration Model via Mixed Linear Transformer and Dual Feature‐Based Knowledge Distillation

Transformer architectures have demonstrated remarkable efficacy in image restoration; however, their quadratic self‐attention complexity imposes severe computational constraints for deployment on resource‐limited platforms. We investigate an alternative pathway within the transformer family itself: replacing quadratic attention with linear attention mechanisms while preserving the architectural strengths of spatial‐wise and channel‐wise feature modelling. We introduce Mixed Linear Transformer (MLT) blocks, each integrating multi‐head spatial‐wise linear self‐attention and multi‐head channel‐wise linear self‐attention, to construct a lightweight student network. To compensate for the representational capacity reduction inherent to linearization, we propose a Dual Feature‐based Knowledge Distillation (DFKD) framework that operates in an offline manner: a high‐capacity transformer‐based teacher is first pre‐trained via supervised learning and subsequently frozen, while the student learns to replicate both spatial‐wise and channel‐wise attention patterns through dedicated feature alignment losses. Extensive evaluation across multiple image restoration benchmarks demonstrates that the distilled MLT‐based model, operating with merely 1.45 M parameters and 12.8 GFLOPs, achieves competitive or superior performance relative to state‐of‐the‐art approaches.