Audio-guided visual selection for efficient multimodal fusion via a discrete variational autoencoder

In multimodal learning, not all parts of all modalities are of equal importance. An additional modality can introduce overlapping signals while requiring a significant increase in computation. As a result, there are many cases where computational requirements for a multimodal model are increasing while its performance is degrading. Early detecting and subsequently fusing only the complementary segments that exist in the high-resource modality by previewing the low-resource modality is how one can overcome this problem. This work proposes a framework to dynamically learn the selection of such salient segments in the vision feature space by previewing the acoustic features of the audio. The framework contains a self-supervised discrete variational autoencoder (dVAE) component and a traditional supervised multimodal fusion component. The former is trained to produce a discrete masked vector in the latent space, as opposed to the traditional continuous latent VAE, by using Binary Concrete distributions as the encoder and the prior distributions. This masked vector removes the non-salient segments in the vision feature space before passing them to the latter component. At inference time, only salient vision features need to be extracted, making the framework very efficient. The empirical experiment on a large-scale video interview dataset reveals that this approach outperforms SOTA in terms of both efficiency and performance, requiring only 15.42 percent of the original vision features (a 23% relative improvement) while improving correlation from 0.6792 to 0.7196 and F1-score from 74.73% to 76.64%.