Automated scoring of constructed responses: BERT performance on near- and far-transfer biochemistry problems

Automated scoring tools have shown promise for assessing student scientific reasoning at scale, but their application to evaluating students’ transfer of learning to new contexts remains understudied. This study examined how well a BERT-based multi-task classification model (Bidirectional Encoder Representations from Transformers, BERT) reproduced expert human scoring of undergraduate responses to near- and far-transfer problems in biochemistry, with a focus on noncovalent interactions. Using responses from 792 students, we trained and evaluated the model across six rubric categories for near-transfer problems and five categories for far-transfer problems. Quantitative results indicated strong model performance in near-transfer contexts (overall accuracy = 0.91) across all categories except for the high-quality claim category. Performance declined in far-transfer contexts (overall accuracy = 0.87), particularly in categories involving evidence use and claims for new interaction predictions, but remained robust overall. Qualitative thematic analysis showed that the model aligned with human raters when student responses used discipline-specific vocabulary and explicit, unambiguous reasoning. Misalignments occurred when responses involved vague claims, alternative but valid reasoning, superficial scientific language, non-normative ideas, or missed contextual information. The model also showed limited sensitivity to graded levels of scientific accuracy due to the use of a three-class output structure. Together, these findings demonstrate the potential of BERT-based models to support automated scoring of near-transfer problems while highlighting key challenges in analyzing student thinking in far-transfer contexts.