Self-regularizing diffusion transformers with uncertainty-aware cross-modal contrastive learning for stable multimodal generative AI

Diffusion models have established themselves as the leading paradigm for high-fidelity generative modeling, yet persistent challenges including training instability, mode collapse, and computational inefficiency continue to limit their practical deployment across diverse real-world applications. We present Self-Regularizing Diffusion Transformers (SRDT), a novel generative framework that addresses these limitations through three core technical innovations: (1) a hierarchical cross-modal contrastive alignment module that jointly optimizes diffusion dynamics and multimodal semantic consistency within a single end-to-end objective, enforcing structural coherence between visual and textual latent trajectories at every denoising timestep; (2) an uncertainty-aware attention mechanism that dynamically suppresses unreliable activations through feature-level variance and entropy estimation, enabling robust generation under varying noise conditions; and (3) a self-regularization term that enforces hidden-state trajectory smoothness across consecutive denoising steps, providing an explicit inductive bias against training instability with theoretical convergence guarantees. SRDT was rigorously evaluated on six benchmark datasets spanning three task domains: text-to-image synthesis (MS-COCO 330 K, LAION-400 M), medical image reconstruction (BraTS 2023, ISIC 2020), and audio generation (AudioSet, VoxCeleb2). Experimental results demonstrate that SRDT achieves an 18.3% reduction in Fréchet Inception Distance (FID = 11.42), a 23.7% improvement in computational efficiency (58.3 GFLOPs, 7.1 GB memory), and superior zero-shot generalization (61.9% average accuracy) over state-of-the-art baselines including Stable Diffusion, DALL·E 2, and Imagen, while using 385 M parameters — fewer than all compared methods. Comprehensive ablation studies confirm the individual contribution of each proposed component, with cross-modal alignment yielding the largest performance gain (FID reduction of 3.31). Statistical significance testing across five independent runs confirms all improvements at p < 0.001 with large effect sizes (Cohen’s d > 1.4). These results establish SRDT as an effective, efficient, and theoretically grounded framework for stable multimodal generative AI.