Multimodal approach predicts sound direction and camera rotation in simulated environments, suggesting effective machine learning without labeled data.
When a person turns their head, two things happen at once: the visual scene shifts, and sounds appear to come from a different direction. This paper explores whether a machine can learn to understand this relationship on its own, without any human-labeled data. We train two models together — one that estimates how much a camera has rotated between two images, and one that estimates where a sound is coming from using two-channel audio. By linking these two models through a shared geometric rule, they teach each other during training. To support this research, we also built a new synthetic dataset of 50,000 audio-visual pairs recorded in simulated indoor environments.
No takes yet. Share an insight, caveat, or question.
Chih-Bo Chen (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: