TVNet: Multimodal Medical Image Fusion by Dual-branch Network with Vision Transformer and One-Shot Aggregation Module

Abstract The task of medical image fusion is to synthesize complementary information from different modals medical images, which is of very significance for clinical diagnosis. The existing medical image fusion algorithms overly rely on convolution operations and cannot establish long-range dependencies on the source images. This can lead to edge blurring and loss of details in the fused images. Because the Transformer can effectively model long-range dependencies through self-attention, a novel and effective dual-branch feature enhancement network called as TVNet is proposed to fusing multimodal medical images. This network combines Transformer and CNN to extract global context information and local information to preserve detailed textures and highlight the structural characteristics in source images. Furthermore, to extract the multi-scale information of images, an enhancement module is used to obtain multi-scale characterization information, and the two branches information are efficiently aggregated at the same time. In addition, a hybrid loss function is designed to optimize the fusion results at three levels of structure, feature and gradient. Experiments results prove that the performance of the proposed fusion network outperforms seven state-of-the-art methods in both subjective visual effects and objective metrics.