Answers are grounded in this paper and cite their sources.
Most medical schools in the United States and Canada evaluate applicants based on both academic achievements and non-cognitive (soft) skills such as professionalism. Commonly, medical school applicants are required to complete Casper—an online constructed-response situational judgment test developed and administered by Altus Assessments to assess various aspects of professionalism such as ethics, empathy, resilience, and collaboration. Examinees writing Casper are given a set of real-life, hypothetical scenarios and asked to explain how they would handle the situation and their reasoning behind it. Since each Casper scenario focuses on multiple aspects of professionalism, examinees do not have to target any specific aspect in their responses. To ensure the correct use and interpretation of Casper scores, the Altus team needs to reliably evaluate whether examinees’ responses for each scenario address the anticipated aspects of professionalism identified by subject mat-ter experts. In this study, we demonstrate how natural language processing (NLP) methods can be employed to characterize the meaning and context of written responses in relation to different aspects of professionalism. Using the Lbl2Vec and BERTopic algorithms, we performed a semi-supervised analysis of written responses from a large sample of medical school applicants (n = 635,106) by clustering the responses into latent topics based on a set of predefined keywords and labels for each professionalism aspect. Our analysis yielded substantive topics that were mostly aligned with the anticipated professionalism aspects. Furthermore, in terms of topical prevalence, some aspects of professionalism (e.g., self-awareness) appeared to be more dominant than the other aspects in the responses. Overall, our proposed approach delivers a feasible mechanism to identify overlapping topics and aspects to maintain test quality and improve human rater training and monitor-ing in constructed-response situational judgment tests.
No takes yet. Share an insight, caveat, or question.
Bulut et al. (2022) studied this question.