Reconstructing missing modalities of magnetic resonance images (MRIs) is a significant challenge in the medical imaging field. Current generative approaches such as generative adversarial networks (GANs) and diffusion models (DFs) have shown promise in synthesizing high-fidelity 2-D slices. However, they fall short in producing high-quality 3-D results due to their inability to leverage the contextual information across adjacent slices, resulting in low-quality images with poor interplane consistency. This problem is exacerbated when trained on datasets obtained with 2-D scanning protocols, where different MRI modalities have varying resolution between slices, leading to blurry and low-resolution results. To overcome these issues, we propose a novel fine-tuning strategy that can enhance the 2-D multimodal synthesis models to improve both consistency and interplane resolution in 3-D images. We begin by developing a novel attention-based module that can effectively empower generative models to produce 3-D results with high consistency between adjacent slices. Furthermore, we incorporate self-supervised super-resolution (SR) to deal with the varying resolution problem and eventually improve the interplane resolution of generated images. Instead of directly applying SR models that may bring about potential artifacts, we design a novel uncertainty modulation strategy, which optimizes the utilization of super-resolved images for high-quality reconstruction. We extensively evaluate our method through supervised and unsupervised multimodal synthesis using different generative models, including GAN- and DM-based models, all of which demonstrate the superiority and flexibility of the proposed method.
Song et al. (Thu,) studied this question.