Answers are grounded in this paper and cite their sources.
In engineering curricula, a final capstone project connects to daily practice in an (as much as possible) authentic setting, to test and prepare students for careers in the capital-intensive chemical engineering field. This paper examines a particular aspect of using capstone engineering design projects for experiential teaching: examination and evaluation of students’ practical work in firms. Such projects are as diverse as they are complex; therefore, performance evaluations of these projects tend to be particularly complicated. These projects do not typically correspond well to a singular set of performance criteria or rubrics, and therefore, a degree of inter-evaluator bias is to be expected. Drawing on a content analysis of the qualitative evaluations of individual design projects, completed by several cohorts of graduates of an engineering program, we uncover a phenomenon, which we call the multi-evaluator paradox of design projects. This phenomenon holds that statistically speaking, all examiners of these projects grade these projects essentially similarly, i.e. there is no significant inter-grader bias of both final and sub-category grades. Simultaneously, however, the verbal motivations of these grades vary greatly between examiners, even when there are structured rubrics underpinning their assessments. Using Computational Text Analysis (CTA), we examine this thus-far undocumented phenomenon and offer possible explanations for its existence. We question whether the existence of this phenomenon is detrimental to educational quality, and we provide suggestions for managing these consequences. • Assessment of chemical process design projects becomes more complex • Computational text analyses of these project assessments show large variation • Examiners use different jargon to justify comparable grades for comparable projects • Alignment of grader’s justification even exists when using shared rubrics
No takes yet. Share an insight, caveat, or question.
Steen et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: