Evaluation of Source Code with Item Response Theory

The analysis of source code produced by novice programmers could provide interesting insights into their learning progress, particularly in introductory programming courses. Yet, as the programming ability of a person is assumed to be quite complex, it is not likely that it would be observable directly in its total. Instead, we regard those abilities as latent psychometric constructs and apply the methodology of item response theory (IRT) to assess their manifestations. In preparatory work, we had identified a list of items that represent the central concepts of object-oriented programming. In this paper we propose a methodology that allows the evaluation of coding abilities by analyzing the application of those concepts. We demonstrate this methodology by exemplarily analyzing source code that was produced during programming projects. The results provide interesting information about the difficulty of the concepts' application and the distribution of the respective coding abilities among the students.