Optimizing Measurement Precision and Diagnosticity for a Two-Dimensional Assessment of Reading Comprehension

This study examined the validity of a computerized adaptive testing (CAT) version of a multidimensional test of reading comprehension that produces two scores: an overall reading comprehension score and a diagnostic classification. The Multiple-choice Online Causal Comprehension Assessment, or MOCCA, uses several features to render scores across two measurement dimensions. We sought evidence that CAT increased testing efficiency by reducing test time and length without sacrificing measurement precision, while also producing better diagnostic classifications and equivalent convergent validity compared to a fixed-item test (FIT). Using a blocked randomized control trial design, we compared CAT and FIT versions on several dependent variables related to testing efficiency, precision, and validity. CAT reduced average test length and testing time while improving measurement precision for the reading comprehension score and improving diagnostic classifications. Results also revealed similar correlations between reading comprehension scores and criterion measures, and evidence of the predictive utility of both scores.