The Effect of Test-taking Familiarity on Standard Error of the Measurement in Computerized Adaptive Tests

Computerized Adaptive Tests (CAT) tailor item difficulty to the examinee's ability, potentially reducing the standard error of measurement (SEM) even while administering fewer items. As one of the main goals of an assessment system is to minimize SEM and to provide ability estimations closest to the true score of examinees, investigating factors related to SEM provides valuable information. As CAT is a comparably new testing system in classroom use, test-taking familiarity might be related to SEM in CAT, especially for younger students. However, most of the CAT studies related to SEM use simulated data and experimental studies on the live application of CAT in classroom setting are missing. Thus, the current study investigates the impact of test-taking familiarity on SEM in a CAT consisting of 4th-grade mathematics items. This experimental study involved 403 students assigned randomly into control and experimental groups. The experimental group received easier warm-up items before the CAT while the control group began the CAT without warm-up items. Results indicated that test-taking familiarity significantly reduced SEM (control group M=0.369; experimental group M=0.342; η2 = .579). The results revealed that students who began with warm-up items had lower SEM. Thus, the current study suggests that using warm-up items in CAT is likely to increase the precision of ability estimations especially for younger students.