Effects of Differentially Time‐Consuming Tests on Computer‐Adaptive Test Scores

Time limits on some computer‐adaptive tests (CATs) are such that many examinees have difficulty finishing, and some examinees may be administered tests with more time‐consuming items than others. Results from over 100,000 examinees suggested that about half of the examinees must guess on the final six questions of the analytical section of the Graduate Record Examination if they were to finish before time expires. At the higher‐ability levels, even more guessing was required because the questions administered to higher‐ability examinees were typically more time consuming. Because the scoring model is not designed to cope with extended strings of guesses, substantial errors in ability estimates can be introduced when CATs have strict time limits. Furthermore, examinees who are administered tests with a disproportionate number of time‐consuming items appear to get lower scores than examinees of comparable ability who are administered tests containing items that can be answered more quickly, though the issue is very complex because of the relationship of time and difficulty, and the multidimensionality of the test.