Using Rasch Wright Map on Web RaschOnline to Assess ChatGPT-Math-Solver’s Proficiency in Mathematics College Entrance Examinations: A Comparative Study (Preprint)

BACKGROUND The ChatGPT-Math-Solver (CGMS) is a specialized tool developed to assist students in solving mathematical problems using the capabilities of the ChatGPT language model. Despite the efficacy of artificial intelligence (AI), there is a lack of comprehensive research assessing CGMS’s competence on Mathematics tests for college entrance examinations using the Wright Map based on Rasch analysis. OBJECTIVE This study aims to (1) showcase the utility of Rasch Wright Map on RaschOnline, and (2) determine the grade achieved by CGMS when compared to a typical sample METHODS CGMS’s capability was evaluated using 18 multiple-choice questions (MCQs) from the Mathematics tests conducted for Taiwan's college entrance examinations in 2024. Using Rasch rating scale model (RSM) with web Raschonline, 300 simulated students with normal distributions were generated to compare with CGMS’s responses based on item difficulties assigned by ChatGPT-4Vo. RaschOnline was used to create Wright map, along with item-difficulty plot, item characteristic curve (ICC), KIDMAP, and performance plot, to address the research objectives. RESULTS The findings revealed the following: (1) CGMS’s capability was shown on the Wright Map, with -0.46 logits (or T-score 54.6) as grade C in overall five grades from A to E; (2) Item 5 was the most difficult, with 1.43 logits on the item-difficulty plot; (3) Item 5 displayed an good fit to the Rasch model with the infit mean square error (MNSQ) of 0.98; (4)An aberrant response with Zscore=2.44 (>2.0, P<.05) on item 6 for one of ChatGPT examinees was shown on KIDMAP; (5) Two testers classified as Grade A, standing out at the top on the performance plot when compared to other counterparts; (6) CGMS's grade would be B or even A if official correct answers are provided for CGMS. CONCLUSIONS Using RaschOnline, this study provides evidence that CGMS possesses the ability to achieve a grade C when compared to a typical sample. After providing the correct answers to CGMS, a 100% correct response rate was then achieved. CGMS's excellent proficiency in answering MCQs in mathematics tests conducted in 2024 for Taiwan's college entrance examinations was evident.