Demystifying a Q-Matrix for Making Diagnostic Inferences About L2 Reading Skills

In researching the potential of cognitive diagnostic assessment, researchers concur that the quality of diagnostic inferences is subject to the extent to which the construct representations based on cognitive skills are theoretically compelling, empirically sound, and relevant to test use. In this paper, I argue that the construction of a Q matrix requires multiple sources of evidence supporting the representation of the construct with well-defined cognitive skills and their explicit links to item characteristics. I illustrate the process of constructing and refining a Q matrix using the results from a large-scale study that examined the validity of applying cognitive diagnosis assessment approaches to LanguEdge reading comprehension tests. I focus on the characteristics of reading skills identified from verbal protocols along with the analyses of text and items and discuss issues related to identifying reading skills and determining the granularity for diagnostic inferences. By demonstrating the process of refining a Q matrix, I discuss fundamental issues arising from the application of cognitive diagnostic assessment through retrofitting. I believe that the paper provides useful guidelines for those who are interested in designing a systematic cognitive diagnostic assessment for L2 reading comprehension abilities. Notes 1Note that the reported model is a reduced version of the full Fusion Model. The full model includes a continuous residual ability parameter intended to capture the influence of skills not included in a Q-matrix. However, most of the Fusion Model applications used the reduced version where the continuous residual ability parameter is removed because of its excessive influence on the item response function (see CitationHenson, Templin, & Douglas, 2007; CitationJang, 2009). 2Minimum three items per skill are recommended (see CitationHartz & Roussos, 2005).