Eye-tracking Study on Students' Reading Process of Multimodal Texts: Focusing on Graphic Novels and Picturebooks

The purpose of this study was to compare and analyze the gaze movement patterns of sixthgrade elementary school students while reading multimodal texts—graphic novels and picturebooks—in order to investigate their reading strategies and meaning-making processes. A crossover experimental design was employed in which the same participants read both types of texts. Quantitative and qualitative data were collected using an eye-tracking device, focusing on gaze path, heatmap, and areas of interest (AOI), and were supplemented by retrospective think-aloud interviews. The analysis of gaze paths revealed that during graphic novel reading, students followed a structured pattern in line with the sequential flow between speech balloons and images, indicating a text-centered information processing tendency. In contrast, during picturebook reading, students exhibited more fluid gaze patterns led by image exploration, highlighting the construction of meaning through the interaction between visual and verbal information. Heatmap analysis showed strong fixation on speech balloon texts and characters’ faces in graphic novels, whereas in picturebooks, fixations were concentrated on text and images marking narrative transitions. In AOI analysis, the text area in graphic novels showed the highest dwell time and number of fixations, while in picturebooks, significant gaze distribution was also observed in image areas related to the text. Interpreting these results through Serafini’s (2012) reader resource theory for successful multimodal text reading, readers of graphic novels primarily activated roles as "navigators" and "designers," whereas readers of picturebooks more prominently exhibited the roles of "interpreters" and "navigators." This study provides an empirical, process-oriented approach to multimodal text reading and offers meaningful implications for reading education and research in multimodal literacy by identifying genre-specific differences in information integration strategies.