Multimodal Affect Classification at Various Temporal Lengths

Earlier studies have shown that certain emotional characteristics are best observed at different analysis-frame lengths. When features of multiple modalities are extracted, it is reasonable to believe that different temporal lengths would better model the underlying characteristics that result from different emotions. In this study, we examine the use of such differing timescales in constructing emotion classifiers. A novel fusion method is introduced that utilizes the outputs of individual classifiers that are trained using multi-dimensional inputs with multiple temporal lengths. We used the IEMOCAP database which contains audiovisual information of 10 subjects in dyadic interaction settings. The classification task was performed over three emotional dimensions: valence, activation, and dominance. The results demonstrate the utility of the multimodal-multitemporal approach. Statistically significant improvements in accuracy are seen for in all three dimensions when compared with unimodal-unitemporal classifiers.