Classifying Confusion: Autodetection of Communicative Misunderstandings using Facial Action Units

Classification of secondary emotions via facial expression analysis or other forms of social signal processing is a topic which has received more attention recently due to the increasing computational and algorithmic power of emotion expression and detection software. Of the social emotions (or affective states), confusion detection is of paramount importance in interactive tasks that require communicating from different viewpoints. Confusion has only fairly recently been the subject of computational modelling approaches owing to the contention as to what expressive components are foundational. One aspect of confusion that has been perhaps de-emphasized in such models is the temporal component. This includes facial expression and non-expression data and the temporal context (state) within which they are embedded. Using the Facial Action Coding schema of Ekman, this article reports the findings of a Long-Short Term Memory (LSTM neural network) modelling approach to (video) instances of confusion as expressed in the context of a “instructor-follower” interactive (map directions) task. The LSTM neural network is able to encode the temporal context of expressed micro instances of positive, negative, neutral and confusion based affective states and is compared against a simpler algorithm used in FaceReader 7.1. that does not account for temporal context (memory state). The LSTM neural network, when trained on time series data of a number of theoretically relevant Action Units, performed at a level comparable to the chosen benchmark highlighting its potential utility. Lesioning trials of the pre-trained best model showed that Action Units 25, 26 and 27, involving open mouth expressions, were the most individually important contributing factors to the model's performance. Testing on novel and larger datasets, as well as possible incorporation of multimodal social signals, would be expected to enhance generalizability and provide potential avenues for future work.