“Am I Understood?”: How the Interplay between Embodiment and Theory of Mind Behavior Affects LLM-Based Conversational Agents on Perceived Trust, Anthropomorphism, Presence, Usability, and User Experience

Large Language Model (LLM) -based conversational agents have become increasingly popular in recent years due to their novel capacity for natural, human-like dialogue interactions. However, mistrust in LLMs persists due to concerns about privacy, the potential for incorrect responses (often referred to as “hallucinations”), and issues related to social bias. Previous AI research shows that anthropomorphic form positively influences users’ perceptions. However, this aspect remains under-explored in LLM-based conversational agent research. Our research features two anthropomorphic forms: embodied and behavioral. Embodied Anthropomorphic Form (EA) encompasses chatbot, chatbot with Text-to-Speech (TTS) , and Embodied Conversational Agent (ECA) interface designs. Behavioral Anthropomorphic (BA) Form involves LLMs instructed with and without Theory of Mind (ToM) principles. In an empirical evaluation, we explored how interplay between BA form and EA form, and vice-versa, affects users’ perceptions of LLM-based conversational agents on trust, anthropomorphism, presence, usability, and user experience. Our findings provide evidence of such effects, offering novel insight into the influence of both anthropomorphic forms on perceived anthropomorphism, presence, usability, user experience, and their positive impact on user trust in LLM-based conversational agents. However, the combined highest (i.e., ECA with ToM behaviors ) and lowest (i.e., Chatbot without ToM behaviors ) levels of both forms result in lower user trust, suggesting a complex relationship between embodiment and ToM behaviors that warrants further investigation.