Work in Progress: Investigating the Effectiveness of an Orchestration Tool on the Nature of Students’ Collaborative Interactions During Group Work

WIP: Investigating the Effectiveness of an Orchestration Tool on the Nature of Students' Collaborative Interactions During Group Work The role of the instructor is characterized by different pedagogies as both "guide on the side" and "sage on the stage." In the context of collaborative problem solving (CPS), a form of pedagogy that has become increasingly common in engineering courses, it is necessary for instructors to reactively provide guidance for students' conceptual understanding while simultaneously supporting their interactions, rather than solely providing content-focused instruction at their chosen pace. This can be done through many means, such as classroom affordances, task structures, and face-to-face instructor interventions and feedback. However, supporting CPS remains challenging, especially for novice instructors (e.g. teaching assistants; TAs) who often teach introductory engineering courses. This work-in-progress study investigates the effectiveness of instructor interventions by evaluating the influence of an orchestration tool deployed while students collaboratively solved ill-structured engineering design tasks in real time. Orchestration technology is developed to support instructors in managing real-time factors within a classroom. Researchers have developed supportive tools that help teachers identify crucial learning moments and provide actionable information. In order to better support TAs, who do not always have the resources or training to facilitate CPS, we developed an orchestration tool that provided strategies to strengthen TAs' interventions with groups and support students' collaborative interactions in real time. We deployed our tool in collaborative discussion sections that were part of a required undergraduate engineering course; these sections were taught by graduate and undergraduate TAs who had little or no experience teaching CPS. In this work, we explore TAs' interactions with the orchestration tool and examine ways in which 1) the tool impacted instructor interventions and 2) instructors interventions impacted students' group work. This study evaluates the results of implementing our tool in three discussion sections held over two weeks. Across these sections, a total of 26 groups of three to four undergraduate engineering students worked to solve collaborative, ill-structured tasks. The tool was provided to all TAs as a resource that enabled them to identify groups needing collaborative support while providing suggestions for how to attend these needs; both interventions that were and were not guided by the tool were analyzed to understand its effectiveness. Our initial findings indicate that when TAs used the tool, they were more likely to monitor a group before intervening and to initiate the intervention themselves, suggesting that the tool effectively directed TAs' attention toward groups. In contrast, without the tool, instructors were more likely to end their intervention by checking the groups' understanding. Our continued analysis will seek to understand how these factors affect groups' collaborative interactions. Our findings respond to calls to implement CPS in engineering courses by understanding if and how orchestration technology aided TAs in facilitating this form of pedagogy. By investigating how an orchestration tool supported TAs and groups of students, our analysis will provide recommendations for how similar tools can be developed to further promote this form of learning.