The patterns of perception of uni- and multimodal expressions of human affective states are studied. Objective: to search for functional mechanisms of cross-modal integration of vocal and facial forms of bimodal (voice + face) expressions. According to the hypothesis, the type of cross-modal relations in the perception of human affective states is determined by the content (category) of the target emotion. The features of perception of 14 unique and bimodal expressions of the same emotional states of actors-sitters were compared. The research was conducted on the platform of the Geneva Emotion Recognition Test GERT. The experiment consisted of three series. In the first, the observers were presented with short audio videos of the affective states of the sitters, in the second the same videos, but without sound, in the third the intonation of the voice without a video image. Independent groups of subjects participated in each series: 72 women aged 2227 5.6. It was necessary to determine the emotional state of the sitter. The accuracy of identifications and the dynamics of the structure of categorical fields were analyzed. Five functional mechanisms of crossmodal integration have been identified, their correspondence to the content of emotions, valence and degree of arousal, as well as the structure of the categorical field. The phenomenon of coherence of unimodal expressions of the same affective content is described. The specifics of the organization of bimodal categorical fields are revealed. According to the results the process of cross-modal integration of unimodal states is not limited to the direct interaction of sensory systems, including visual and acoustic. The important thing is the subject (categorical) content and logic of combining diverse impressions about the human condition.
V.A. Barabanschikov (Sun,) studied this question.