Evaluating computer science students reading comprehension of educational multimedia-enhanced text using scalable eye-tracking methodology

Abstract In this research, a mixed-method approach was employed to conduct large-scale eye-tracking measurements, traditionally associated with high costs and extensive time commitments. Utilizing consumer-grade webcams in conjunction with open-source software, data was collected from an expansive cohort of students, thereby demonstrating the scalability and cost-effectiveness of this innovative methodology. The primary objective of this research was to discern the disparities in reading behaviour when students were presented with standard text accompanied by illustrations, compared to the same text with highlighted key terms. The participants, comprised of first-year university students, completed a questionnaire and an introductory test to ascertain their knowledge level. Subsequently, they were segregated into two groups and participated in two reading sessions, during which their ocular movements were recorded. The amassed data underwent both qualitative analyses, facilitated by visualizations, and quantitative analysis, employing statistical measures on the data and test results. Notably, no significant difference was observed in the gaze patterns or test results between the experimental and control groups. However, a significant divergence in gaze patterns was identified between high-achieving students and those experiencing difficulties, as evidenced by the averaged composite heatmaps generated from the data. The findings underscore two pivotal points. Firstly, the feasibility of conducting large-scale eye-tracking experiments is demonstrated. Traditional studies in this field often employ small population samples due to the time and financial constraints associated with methods that utilize specialized eye-tracking hardware. In contrast, our methodology is scalable, relying on low-end hardware and enabling students to record data on their personal devices. Secondly, while eye-tracking may not provide substantial benefits for fine-tuning text already optimized for readability, it could serve as a valuable tool for identifying and assisting learners who are struggling. This mixed-method approach holds significant potential to revolutionize the conduct and interpretation of eye-tracking studies within educational settings.