Evaluation of Humans and Software for Grading in an Engineering 3D CAD Course

In skill-building courses such as an introductory 3D CAD course, instructors typically provide many assignments for students to practice and improve their 3D modeling skills.Frequent and accurate assessments give students the opportunity to identify errors and address deficiencies more efficiently, promoting quicker acquisition of the skill.In an ideal learning environment, students would be provided feedback at every class meeting, but that can be a daunting task as grading 3D CAD homework is difficult and time-consuming.The objective of this work is to compare human and software grading of student's 3D CAD files and quantify the speed, accuracy, and effectiveness each.A statistical analysis was performed on 5200 models from three different assignments to compare the two modes of grading.Better understanding of the different grading practices enables resource allocation based on strengths of humans and computers; resulting in a more efficient combination of resources.The results show that Graderworks software (GW) was more accurate and repeatable than human graders (TAs) in quantitative comparisons: evaluating material, volume, shape, and sketches.TAs often made a few clerical errors per assignment that limited the effectiveness of the file management structure and subsequent calculations from manually entered fields like the name or username.However, a single change in the learning management software naming convention of files lead to a large scale clerical error with similar frustrations with automation of grading.Still, one of the biggest challenges we have experienced with human grading is the high variability in speed and accuracy of graders; an ANOVA test showed that error rates differ between TAs at a statistically significant level.TAs are effective at providing informative feedback that provides direction for improving the model, but it is a time consuming process.At this time, the software is not able to offer substantial and specific feedback to the students on how to improve, and it is recommended to use the computational grading tools in conjunction with human graders.Using the software to prioritize which files need TA feedback, those with similarity scores below a threshold value, may lead to a more efficient and effective use of resources to provide a quality feedback loop.