From item writing to item completion: investigating multiple-choice reading test items through item writer’s intentions and test-takers’ reported processes

Writing multiple-choice (MC) test items that accurately target specific reading constructs remains challenging and time-consuming. Despite careful item development, what test developers intend an item to measure may not correspond to the processes test-takers actually use when answering it. This exploratory study documented the original item writer’s option-level intentions when constructing MC items and examined the extent to which these intentions were corroborated by test-takers’ retrospective verbal reports of their test-taking processes. The documentation revealed that the relevant text portions and cognitive activities intended for each option within an MC item may vary. Triangulation of item writer intentions and reported test-taking processes showed stronger convergence for relevant text portions than for cognitive activities. Divergences between the two data sources were largely associated with test-taking strategies employed by participants. Importantly, documenting the item writer’s option-level intentions provided grounded explanations for why test-takers appeared to engage in different processes when selecting different options. These findings offer a new methodological direction for construct validation of MC items and suggest implications for the future development and evaluation of multiple-choice items in reading assessment.