Still Not Clear? Exploring the Impact of Clarifying Assessment Items on Assessor Cognition in Medical Education.

Assessment variability in formative assessment occurs when assessors observing a trainee performing the same task evaluate the trainee differently. One major contributor is uncertainty regarding assessment criteria, and efforts to clarify criteria are not always successful. This study explores the cognitive processes that occur in assessors' minds when assessment criteria are clarified. We interviewed clinical teaching faculty from one residency program in a single institution regarding their perceived expectations of select assessment items before and after providing clarifying criteria and how the clarification changed their perception. We analyzed the data thematically. Assessors' cognitive interaction with assessment clarification is a function of four factors: 1) Assessors' fixed ideation, 2) Content of the criteria themselves, 3) Context and setting of criterion interpretation, and 4) Interaction between the assessor and the trainee. The cognitive effects of clarifying assessment items depend not only on the assessor and criteria but additionally on their interactions within a professional and academic context. The complexity and multifactorial nature of assessment variability may explain the difficulty in mitigating criterion uncertainty.