Task Complexity, an Inner Speech Analog, and Prompt Design Affect GPT-3.5's Response Behavior in the Two-Step Task

Large Language Models (LLMs) have shown impressive performance across a wide range of tasks, yet the mechanisms behind their response behavior remain poorly understood. Drawing on insights from cognitive science and the computational framework of model-free and model-based reinforcement learning (RL), we examined conditions under which GPT-3.5 can exhibit behavioral signatures of model-based RL in the two-step task, a well-established paradigm for studying human learning and decision-making. Across nine simulation studies, we varied the task complexity and prompt design to examine conditions under which previously reported model-based RL signatures could be replicated. We found that GPT-3.5 showed model-based RL signatures in simplified task settings, but that these effects disappeared under de-simplified task versions. Crucially, model-based reasoning signatures could be re-established when incorporating an inner speech-analog prompt, requiring the LLM agent to reflect on previous trials and anticipate consequences of future actions. Our fine-grained analyses of behavioral learning dynamics highlight the role of structured reflective and anticipatory linguistic processes in enabling complex response patterns in LLMs and reveal both similarities and differences to human decision-making patterns.