Leveraging Concept Annotations for Trustworthy Multimodal Video Interpretation through Modality Specialization

Multimodal datasets usually come as multimodal data annotated for a certain construct (such as depression). However, for such tasks of video interpretation, models must not only make accurate predictions, but make them for the right reasons. Ensuring model trustworthiness is however hampered by the lack of per-modality information. We consider the case of a recently introduced dataset for the video interpretation task of detecting objectification, annotated for this end task along with multimodal explanatory concepts that provide per-modality labels. With such additional knowledge, we study how to design models with both high task accuracy and modality trustworthiness. We first introduce the MSpecF framework articulating and fusing a spectrum of variably specialized models, and two trustworthiness metrics. We show that modality-specialized models generally maximize trustworthiness, and maximize task accuracy for confident modalities. For less certain modalities, task accuracy is maximized by non-specialized models. We show that the full fusion of specialized models MSpecF(All*) achieves advantageous trade-offs between task accuracy and trustworthiness compared to other fusion choices. This work shows that rich per-modality annotations of moderate-size datasets allow to make more trustworthy models, essential for applications such as supporting social scientists in analyzing complex social constructs.