Comparing Human and ChatGPT Performance on the Force Concept Inventory: An Item-Level Analysis Through Rasch Modeling and the Theory of Conceptual Fields

This exploratory study compares the conceptual performance of human respondents and ChatGPT on the Force Concept Inventory (FCI), a widely used assessment designed to diagnose alternative conceptions in Newtonian mechanics. Rather than relying exclusively on overall performance scores, the study investigates whether item-level analyses reveal qualitative differences between human and AI responses. The dataset comprised responses from fourteen Belgian pre-service physics teachers and eleven independent ChatGPT sessions generated through different user accounts. A Rasch model was employed to estimate respondent proficiency and item difficulty, complemented by an item-by-item comparison of success rates and analyses of Item Characteristic Curves (ICCs). The findings indicate that although ChatGPT achieved an overall level of performance comparable to that of several human participants, substantial discrepancies emerged on conceptually demanding items requiring the coordination of multiple physical relationships. These discrepancies are interpreted through Vergnaud’s Theory of Conceptual Fields as suggesting that successful human performance may involve richer coordination of operational invariants than can be inferred from ChatGPT response patterns. Because of the exploratory design, the small sample size, and the asymmetry between human and AI testing conditions, these interpretations should be regarded as hypothesis-generating rather than confirmatory. The study illustrates how item-level analyses, combined with a conceptual framework grounded in physics education research, may provide a more informative perspective on human–AI comparisons than global scores alone.