A hybrid transformer and Mamba network with mixture of experts for deformable multimodal 3D medical image registration

Deformable 3D multimodal medical image registration is a fundamental yet challenging task that requires accurate alignment of volumetric image pairs acquired from different imaging modalities, such as MRI and CT or MRI and TRUS, where large appearance differences, structural variability, and geometric misalignment make correspondence estimation particularly difficult. We propose DH-TMReg, a weakly supervised deformable cross-modality hybrid registration framework that integrates a Mixture-of-Experts (MoE) bottleneck to achieve accurate and robust 3D multimodal volumetric alignment. The architecture couples a Swin Transformer (ST) branch processing the fixed image and a Mamba Vision (MV) state-space model branch processing the moving image through a sequential cross-fusion strategy that progressively enriches each modality with complementary representations from the other across four hierarchical levels. To address the inherent geometric inconsistency between unregistered multimodal 3D image pairs, Deformable Cross-modal self-attention Fusion Skip Connections (DCASC) are applied at every encoder level to adaptively realign and fuse spatially inconsistent dual-branch features through learned deformable sampling offsets. At the bottleneck, a MoE module with top-2 gating over 8 expert networks dynamically routes features through specialized sub-networks to increase representational capacity without proportional computational overhead. Extensive experiments on two challenging 3D multimodal registration benchmarks demonstrate the effectiveness of the proposed method, achieving a mean Dice score of 0.837 ± 0.077 and ASSD of 3.386 ± 1.696 mm on the prostate MRI/TRUS dataset, and a mean Dice score of 0.9451 ± 0.0079 and ASSD of 0.916 ± 0.286 mm on the abdominal MRI/CT dataset, outperforming state-of-the-art multimodal registration methods on held-out test data.