Enhance ecological validity of scenario-based experiments on human–AI collaboration: evidence from an interactive LLM-based research tool

Purpose This paper questions whether traditional experiments accurately reflect decision-making collaborations with AI and addresses this challenge, proposing a novel approach to conducting scenario-based experiments in human–AI collaboration. Design/methodology/approach This paper presents the managerial interaction and response to artificial intelligence, a tool for researching managers' interactions with AI, addressing the shortcomings of traditional vignette approaches. It utilizes advanced LLMs combined with a reasoning-and-action framework to improve contextual accuracy, interactivity and ecological validity. It also tests the tool's internal validity by applying precision, recall and F1 metrics and ecological validity, comparing it to vignettes tested by 222 mid-level marketing managers. Findings Evaluation results show that the tool enables participants to engage naturally with AI by integrating experimental variables within a controlled yet dynamic chatbot setting, maintaining both experimental precision and realism. Originality/value This tool marks a promising advance in AI-manager collaboration research, supporting longitudinal research and cross-cultural validation.