Human-AI Interaction in Large Language Model-Integrated Building Energy Management System: User Prompt Strategy

This study investigates how users formulate and utilize prompts when interacting with large language model (LLM)-integrated building energy management system (BEMS), with the objective of understanding their interaction strategies. Prior studies often relied on (1) rigid or pre-scripted user input interfaces in BEMS due to the limited capabilities of early language models or (2) posed single-turn user prompts to evaluate the performance of LLM-integrated BEMS. To address these limitations, this study conducted an experiment where participants were tasked with identifying five energy-saving behavioral changes while interacting with an LLM-integrated BEMS. This experimental setup enabled us to examine how users would naturally formulate prompts throughout their interactions with the system. In total, we collected 156 prompts (comprising 252 sentences) from 20 participants and classified them via six prompt strategies derived from decision support systems (DSS) theory. Our key takeaways were the following: (1) participants intuitively understood core principles of DSS, such as goal articulation, information seeking, and solution evaluation; (2) most participants began their interactions with GPT by directly articulating their goals, without offering any analytical guidance, showing a strong reliance on GPT's analytical capabilities; (3) participants were less engaged with solution evaluation, alternative seeking, and decision-making support, possibly owing to the domain knowledge required for such strategies. This study provides insights into the human side of human-AI collaboration by analyzing the content in prompts, demonstrating how users would communicate their needs and expectations using LLMs for building energy management tasks.