Development of a Multidimensional Computerized Adaptive Testing Program for Measuring Digital Competency Level in Higher Education

Background and Objectives: Thailand has implemented policies to promote and develop digital competencies among students, which require all universities to assess students’ digital competencies prior to graduation. As a result, many small universities face resources shortages for conducting such assessments. Therefore, the development of an online testing system serves as a potential solution to this issue. This research aimed to develop a digital competency item bank and validate the effectiveness of a Multidimensional Computerized Adaptive Testing (MCAT) program. Methodology: This research employed a research and development (R&D) design. Participants consisted of fourth-year undergraduate students enrolled in higher education institutions in Thailand during the academic year 2023–2024, from the fields of Health Sciences, Science and Technology, and Humanities and Social Sciences. The results indicated that the digital competency item bank contained 211 items and 30 indicators. The item bank was developed using a three-parameter Multidimensional Item Response Theory (MIRT) model. The test demonstrated an EAP reliability of 0.865. Digital competency levels were interpreted using a Wright Map. Additionally, the MCAT program was developed using PHP and MySQL. Results: The quality of 211 items in the item bank was analyzed using three-parameter item response theory. The items demonstrated an average discrimination parameter of 1.44, an average difficulty parameter of 0.24, and an average guessing parameter of 0.13. The item bank was categorized into three modules based on the difficulty parameter. The digital competency model was developed in accordance with the between-items multidimensional item response theory with three parameters. Digital competency level cutoff scores were determined by defining a criterion zone on the Wright map at the mean ability threshold (θd) of -0.39. The results indicated that 69.27 percent of students met the digital competency criteria. The highest level of digital competency was observed in Component 1, Information and Digital Media Literacy, followed by Component 3, Adaptability to the Digital Transformation Era, while the lowest level was found in Component 2, Skill for using, developing, and problem-solving with digital tools. In addition, the findings demonstrated that the MCAT program accurately evaluated digital competency with only 22 items, using a variable-length termination criterion of SE(θ) < 0.30 (RMSE = 0.112). This approach provides an effective method to reduce the testing burden. Discussions: The MCAT program demonstrates the feasibility of implementing a reliable and efficient assessment tool in resource-constrained higher education institutions. At the same time, the study suggests a direction for future development in competency assessment on a web-based platform, which enhances the possibilities for implementing MIRT within an educational organization. Conclusions: The MCAT program demonstrated strong performance across all assessed dimensions, including content validity, indicators, and program structure. The MCAT program consists of an item bank, initial item selection, subsequent item selection, and a termination criterion. The MCAT approach requires less time and fewer items than traditional tests, with the optimal number of items for accurate ability estimation being 22. Overall, the MCAT program provides an appropriate and practical solution for equitable implementation across large, medium, and small universities.