seg2med: a bridge from artificial anatomy to multimodal medical images

Abstract Objective. We present seg2med (segmentation-to-medical images), a modular framework for anatomy-driven multimodal medical image synthesis. The system integrates three components to enable high-fidelity, cross-modality generation of CT and MR images based on structured anatomical priors.Approach. First, anatomical maps are independently derived from three sources: real patient data, XCAT digital phantoms, and anatomies-synthetic subjects created by combining organs from multiple patients. Second, we introduce PhysioSynth, a modality-specific simulator that converts anatomical masks into imaging-like prior volumes using tissue-dependent parameters (e.g., HU, T1, T2, ρ) and modality-specific signal models. It supports simulation of CT and multiple MR sequences, including GRE, SPACE, and VIBE. Third, the synthesized anatomical priors are used to train 2-channel conditional denoising diffusion probabilistic models (DDPMs), which take the anatomical prior as a structural condition alongside the noisy image, enabling it to generate high-quality, structurally aligned images within its modality.Main results. The framework achieves a Structural Similarity Index Measure (SSIM) of 0.94 ± 0.02 for CT and 0.82 ± 0.12 for MR images compared to real patient data, and 0.78 ± 0.04 FSIM for simulated CT from XCAT. The generative quality is further supported by a Frechet Inception Distance (FID) of 3.62 for CT synthesis. In modality conversion tasks, seg2med attains SSIM scores of 0.91 ± 0.03 (MR→CT) and 0.77 ± 0.04 (CT→MR).Significance. In anatomical fidelity evaluation, synthetic CT images achieve a mean Dice coefficient exceeding 0.90 for 11 key abdominal organs, and over 0.80 for 34 of 59 total organs. These results underscore seg2med’s utility in cross-modality image synthesis, dataset augmentation, and anatomy-aware AI development in medical imaging.