Embedding virtual anthropomorphic characters as embodied conversational agents (ECAs) in virtual reality (VR) applications offers benefits across various domains, including training environments and therapeutic settings, by simulating face-to-face interaction partners for human users. However, ensuring that ECAs are perceived as authentic and human-like remains a challenge, requiring the integration of multiple modalities to create convincing virtual humans. This thesis explores key modalities that contribute to the believability of ECAs, which act inherently multimodal when delivering speech acts encompassing verbal as well as co-verbal behavior---such as gaze direction and gestures. We investigate the impact of voice and prosody, making suggestions on how to balance technical effort with their effects on perceived social presence. Additionally, we examine auralization, evaluating whether simulating natural sound directionality enhances conversational realism and determining the necessary level of technical fidelity. Furthermore, we delve into some communicative functions conveyed by co-verbal behavior. Specifically, we examine turn-taking, which governs speaker transitions in multi-party interactions, and back-channeling, which conveys agreement or understanding. Throughout this work we put a particular focus on perceived social presence, the extent to which users feel they are interacting with a real person. To this end, we first review existing subjective and objective metrics for measuring social presence, identifying a gap in objective evaluation methods. Therefore, we assess objective metrics for social presence by leveraging the heard text recall (HTR) paradigm, developed in collaboration with psychology researchers. By systematically degrading ECA performance quality using a forementioned co-verbal components, we investigate HTR as a potential proxy for measuring cognitive load and social presence more rigorously. To facilitate VR-based user studies, we introduce and assess the StudyFramework, a newly developed tool that streamlines factorial-design experiments and includes a system for rendering participant avatars to enhance immersion. Additionally, we explore methods for generating and capturing gestures using off-the-shelf VR hardware and analyze their influence on perceived social presence. In summary, this research advances the understanding of ECA behavior in verbal communication, providing insights into key modalities that enhance natural and immersive interactions in VR.
J. Ehret (Wed,) studied this question.