The Justification Effect on Two-Tier Multiple-Choice Exams

Abstract Booming enrollment in computer science has raised the need for efficiently gradable assessments, among which are Multiple-Choice Question (MCQ) assessments. MCQs have drawbacks, among which are: random guessing, limited instructor insight into conceptual misunderstandings, student frustration at not being able to explain for partial credit. Recent years have seen several MCQ refinements that feature a two-tier structure that elicits justifications in addition to a correct-choice answer. These justifications can themselves be structured as choices to enable rapid grading. This paper introduces an additional refinement to the two-tier MCQ to acquire more information about a student's understanding by also eliciting explanations for why the wrong answers are wrong. In addition, this paper introduces a simple new metric, the Justification effect, or J-effect, that is easy to extract and apply in order to detect students that need help and identify questions that have design issues. The use of this approach allows instructors to easily provide automated yet rich feedback to students and to pinpoint issues with test implementations, all while remaining easy to design, implement, answer and grade. The study explores the testing and implementation of three different trials over two semesters.