Sound-to-Image Translation Through Direct Cross-Modal Connection Using a Convolutional–Attention Generative Model

Sound plays a fundamental role in human perception, conveying information about events, objects, and spatial dynamics that may not be visually accessible. However, current technologies such as Acoustic Event Detection typically reduce complex soundscapes to textual labels, often failing to preserve their semantic richness. This limitation motivates the exploration of sound-to-image (S2I) translation as an alternative connection between audio and visual modalities. Unlike multimodal approaches guided by intermediary constraints during the learning process, we investigate S2I translation without class supervision, cluster-based alignment, or textual mediation, a paradigm we refer to as direct S2I translation. To the best of our knowledge, apart from our previous work, no prior study addresses S2I translation under this fully direct setting. We propose a convolutional–attention generative framework composed of an audio encoder and a densely connected GAN integrating self-attention and cross-attention mechanisms. The attention-based model is systematically compared with a purely convolutional baseline. Results show that introducing attention at early stages of the generator significantly improves translation performance, increasing the likelihood of producing interpretable and semantically coherent visual representations of sound. These findings indicate that attention strengthens semantic correspondence between audio and vision while preserving the fully direct nature of the translation process.