Is automated grading of models effective?

Learning how to model the structural properties of a problem domain or an object-oriented design in the form of a class diagram is an essential learning task in many software engineering courses. Since the grading of models is a time-consuming activity, automated grading approaches have been developed to assist the instructor by speeding up the grading process, as well as ensuring consistency and fairness for large classrooms. This paper empirically evaluates the efficacy of one such automated grading approach when applied in two real world settings: a beginner undergraduate class of 103 students required to create an object-oriented design model, and an advanced undergraduate class of 89 students elaborating a domain model. The results of the experiment highlight a) the need to adapt the grading strategy and strictness to the level of the students and the grading style of the instructor, and b) the importance of considering multiple solution variants when grading. Modifications to the grading algorithm are proposed and validated experimentally.