Sound Matters - On the Influence of Audio Renderings on Socio-Cognitive Processing, Presence, and Fear in Social Situations in Virtual Reality

Humans are inherently social beings, and social cues such as faces and voices guide attention and behavior. Auditory perception, especially binaural hearing, is essential for social cognition, enabling sound localization and speech comprehension in noisy environments. Deficits in auditory processing can impair social functioning, and conditions such as social anxiety are linked to reduced social functioning. Since social functioning is closely linked to overall well-being, improving social behavior represents a key objective in psychological research. Virtual reality (VR) is increasingly used to study social behavior due to its flexibility and ecological validity. However, users often report limited social presence, reducing the effectiveness of VR-based interventions especially for social anxiety. One reason may be the dominance of visual over auditory realism: audio is often presented in mono or stereo, reducing naturalness and presence. Binaural auralizations, which provide realistic, externalized spatial audio, may enhance presence and support virtual social interactions. This thesis pursues four main research objectives: identifying suitable behavioral and subjective measures for evaluating binaural realism; assessing immersion, realism, and audio quality across auralization techniques; comparing synthetic and natural speech in a socially stressful VR scenario; and examining effects of binaural audio on affect, presence, and attention under varying social stress levels. Study 1 examined how the virtual visual scene and measurement method affect localization and distance perception of physical sound sources. Across two experiments (N=60), audiovisual incongruence reduced localization accuracy but did not affect presence or realism. Distance estimation was influences by the interaction of task and scene: overestimation increased when using a placement task in a reduced-visibility scene. Study 2 compared localization accuracy for loudspeakers and four virtual audio renderings using a placement task and a gaze-based paradigm (N=49). Binaural renderings produced slightly lower localization accuracy but similar ratings of social presence and realism. A simple generic rendering performed as well as more complex ones. Only the anchor condition lacked externalization and was inferior across measures. Social presence and subjective realism were strongly correlated. Study 3 compared AI-generated text-to-speech with natural human speech in the Trier Social Stress Test (N=40). Both conditions elicited substantial stress responses and produced similar presence and affect ratings, demonstrating the practicality of synthetic speech in virtual social interactions. Study 4 investigated audiovisual realism in a virtual social stress scenario (N=78). A high-stress group showed stronger physiological and subjective stress responses than a low-stress group. Binaural audio increased perceived realism and externalization but did not affect social presence, stress responses, or gaze behavior. High arousal across all groups may have masked audio effects. Across all 4 studies, social anxiety did not consistently affect auditory perception or presence but influenced affective states and subjective evaluations of the interaction. Overall, the findings highlight the importance of VR-specific auditory perception and the role of acoustic immersion. Auditory realism enhances social and physical presence, though its impact varies by context. It appears most effective in low- to moderate-arousal scenarios and may be less critical in highly affective VR applications such as anxiety treatments. Practical advancesn such as TTS integration and simplified binaural rendering methods can support the broader use of realistic audiovisual VR environments in psychological research.