Perception vs reality: a multi-level framework for analyzing user perceptions of LLM-based chatbots

Purpose This study develops a multi-level framework to examine user perceptions of large language models (LLMs). By investigating the relationships among scenario characteristics (irreplaceability, audience and purpose), user perceptions (familiarity, trust and attitudes) and usage contexts (individual, organizational and societal), this study explores how users evaluate LLM-based chatbots in various contexts, addressing a gap in existing research on LLM applications and their user acceptance. Design/methodology/approach Utilizing a mixed-methods approach, this study crafted eight diverse usage scenarios spanning niche entertainment to significant production support. A questionnaire survey captured users’ familiarity, trust and attitudes, enabling a systematic analysis of user perceptions at individual, organizational and societal levels. Hierarchical clustering and linear regression linked user perceptions with scenario characteristics, forming the basis of the proposed multi-level framework. Findings Through the developed framework, this study identifies distinct perception patterns that systematically align with individual, organizational and societal levels of application. For instance, personal assistance scenarios receive higher perception scores, while societal-level applications reveal mixed user perceptions. These insights suggest that enhancing system performance and addressing user concerns are crucial for effective adoption. Originality/value This study introduces a user-centric, multi-level framework for analyzing LLM perceptions across contexts. It highlights the critical role of user perspectives in shaping ethical, reliable and widely accepted LLM applications, offering practical guidance for developers, organizations and policymakers.