“Eye” do not understand! Analyzing the roles of negation and reading comprehension ability in comprehension difficulties with questionnaire items using eye tracking

To mitigate response biases, some questionnaires contain negated items. While previous findings suggest that negated items may increase processing difficulty compared to affirmative items, this has not yet been analyzed among participants with varying reading comprehension abilities. Additionally, it is unclear whether questionnaire respondents are aware of these difficulties. This study investigated the roles of item polarity and reading comprehension ability in item processing using eye tracking and cued-retrospective interviews. Forty-nine adults read and responded to 16 affirmative and 16 negated items while their gaze was recorded. Among others, results showed effects of item polarity and reading comprehension ability on number of fixations and total fixation duration for both the item statements and the response scale but no interaction effects. Interview data revealed a negative relationship between reading comprehension ability and the perceived comprehensibility of negated items. These findings underscore the importance of considering item and reader characteristics when designing questionnaires.