Spatial-Spectral Mixing Transformer With Hybrid Image Prior for Multispectral Image Demosaicing

The snapshot multispectral imaging system using a multispectral filter array (MSFA) efficiently captures sample the multispectral image (MSI) information of scenes and obtain spectral mosaic images. To obtain the complete MSI information from these spectral mosaics, effective demosaicing methods are essential. Traditional MSI demosaicing techniques depend on pixel correlation and various hand-crafted priors, while deep learning-based approaches learn the mapping between spectral mosaic images and MSI directly. However, current methods often fall short in recovery performance, leaving significant room for improvement in the MSI demosaicing field. In this paper, we propose a novel MSI demosaicing method based on the spatial-spectral mixing transformer with hybrid image prior, named SSMT-HIP, to enhance image reconstruction and detail recovery. Our framework, the spatial-spectral mixing transformer (SSMT), is designed to comprehensively learn the spatial-spectral correlations of the data, addressing the limitations of current CNN-based methods in capturing both spatial and spectral characteristics of MSI. Furthermore, we introduce the deep hybrid image prior (HIP), which combines the deep Gaussian scale mixture (GSM) prior and the deep nonlocal auto-regressive (NAR) prior. This hybrid prior is learned in an end-to-end manner through the deep unfolding network. The GSM prior excels at recovering image textures and details, while the NAR prior enhances long-range dependencies in MSI. Extensive experiments on both synthetic and real-world data demonstrate that our proposed method outperforms existing state-of-the-art MSI demosaicing methods.