Measurement precision and user experience with adaptive versus non-adaptive psychometric tests

Research on computerized adaptive testing (CAT) has provided evidence of the psychometric advantages of CAT over traditional fixed-item tests (FIT). However, evidence on test-taking experience is limited. This study investigated the psychometric and psychological impacts of CAT in the context of a multidimensional fluid reasoning test. In an online experimental study, we tested 286 participants, randomly assigning them to either the CAT or FIT conditions. We used two types of measures to assess participants' experience: self-report and time-based measures. The results showed that CAT outperforms FIT in terms of measurement precision but had a minimal impact on the test-taking experience. At the same time, FIT resulted in a more optimistic evaluation: when completing FIT, participants believed they had answered more items correctly than when completing CAT. Yet response time analysis indicates that participants invested more effort when working in CAT. Rapid guessing increased as the test progressed in both CAT and FIT, particularly in the FIT condition with the most difficult items. Providing information about how CAT works or starting the test with easier items may potentially enhance test-takers' experiences.