Chained computerized adaptive testing for the Force Concept Inventory

Although conceptual assessment tests are commonly administered at the beginning and end of a semester, this pre-post approach has inherent limitations. Specifically, education researchers and instructors have limited ability to observe the progression of students’ conceptual understanding throughout the course. Furthermore, instructors are limited in the usefulness of the feedback they can give to the students involved. To address these challenges, we propose an alternative approach that leverages computerized adaptive testing (CAT) and increasing the frequency of CAT-based assessments during the course, while reducing the test length per administration, thus keeping or decreasing the total number of test items administered throughout the course. The feasibility of this idea depends on how far the test length per administration can be reduced without compromising test accuracy or precision. Specifically, the overall test length is desired to be shorter than when the full assessment is administered as a pretest and a subsequent post-test. To achieve this goal, we developed a CAT algorithm called . This algorithm sequentially links the results of each CAT administration using collateral information. We developed the Chain-CAT algorithm using the items of the Force Concept Inventory (FCI) and analyzed its efficiency through numerical simulations. We found that collateral information significantly improved the test efficiency, and the overall test length could be shorter than that of the pre-post method. Without constraints for item balancing and exposure control, simulation results indicated that the efficiency of Chain-CAT is comparable to that of the pre-post method even when each CAT administration includes only 5 items and the CAT is administered 9 times throughout the semester. However, when these constraints are imposed, we found that the efficiency of the Chain-CAT with a test length of five items is lower than that of the pre-post method. Based on this result and analysis of which items were administered, we suggest expanding the FCI item bank by creating additional items or incorporating items from other research-based assessments that have high discrimination parameters and exhibit sufficient variability in their difficulty parameters.