SyncDIT: audio-visual aligned video generation with audio synchronization feature

Audio-driven video generation has recently shown significant progress in producing realistic visual content. However, most previous studies focus primarily on speech or other narrow-band audio sources, limiting their effectiveness in scenarios involving complex auditory inputs, such as musical performances that combine transient sounds with sustained tones. To overcome these constraints, we introduce SyncDiT, a novel framework designed for synchronized audio-visual generation from complex audio signals and reference images. At its core, SyncDiT leverages an audio encoder to capture intricate acoustic characteristics within an audio clip, while an Audio Sync Cross-Attention module aligns video latents with frame-level audio synchronization features. Additionally, we construct the Musical Instrument Video (MIV) dataset, comprising diverse high-resolution instrument performance videos to support this task. Extensive experiments demonstrate that SyncDiT achieves superior audio-visual synchronization and enhanced visual quality compared to existing approaches, underscoring its effectiveness in generating lifelike videos from complex audio inputs.