Abstract The generation of photorealistic human faces utilizing multi-modal inputs presents significant challenges, as existing methodologies like DiffusionCLIP are limited to text-based directives and often struggle with precise attribute control and cross-modal consistency. This paper presents ClipFaceFusion, a diffusion-based framework that amalgamates multi-signal conditioning (text, audio, reference images) with explicit semantic control signals (age and emotion) to produce and alter photorealistic faces. Proposed approach presenting a trainable multi-signal fusion module in conjunction with novel consistency loss functions that provide audio-visual alignment and precise age/emotion regulation within a cohesive Denoising Diffusion Implicit Models (DDIM) framework. Specialized loss functions for age and emotion consistency, along with a multi-tiered identity preservation system utilizing ArcFace, perceptual loss, and reference image alignment, ensure precise attribute regulation and identity conservation. Experiments demonstrate that ClipFaceFusion outperforms leading techniques such as DiffusionCLIP and StyleCLIP in generating realistic faces with precise age and emotional expressions, while facilitating dependable image modification. The framework facilitates applications in media creation, psychological simulations, historical facial reconstruction, and interactive virtual environments by providing superior Cross-Modal Coherence (CMC) and reduced visual artifacts. ClipFaceFusion seamlessly integrates multi-modal inputs into a unified diffusion-based model, establishing a new benchmark for personalized face synthesis and manipulation.
Jiang et al. (2025) studied this question.