What Makes Learning with GenAI Effective? Text Support and Deeper Engagement are Associated with Higher Performance in an Ecology Context

Generative artificial intelligence (GenAI) tools such as ChatGPT are increasingly used for learning, yet evidence remains limited regarding their effectiveness in science learning contexts and how learner–AI interaction relates to performance. In a randomized experiment, 185 university students worked on task sheets about lake eutrophication and restoration. Depending on condition, they could use two informational texts (text-only), ChatGPT configured to answer only on the basis of these texts (ChatGPT-only), or both (text + ChatGPT). Afterwards, all students completed a knowledge test without access to any materials. Test performance was highest in the text-only and text + ChatGPT conditions and lowest in the ChatGPT-only condition, yielding a robust overall condition effect. Chat logs from the ChatGPT conditions were analyzed to examine interaction patterns associated with performance. Although ChatGPT-only students produced substantially more interaction turns than students with access to both texts and ChatGPT, interaction frequency was unrelated to test performance. Instead, specific interaction patterns were differentially associated with performance: deepening questions were positively associated, whereas direct acceptance of AI-generated responses was negatively associated with test performance. Output-steering and questioning additionally emerged as positively associated with performance in individual analyses. Prompting behavior varied with task difficulty, with deepening questions increasing for more demanding ecology tasks. The findings suggest that, in science learning contexts, the educational value of GenAI may depend less on access to the technology than on learners’ epistemic engagement with it.