Answers are grounded in this paper and cite their sources.
Empirical approach identifies SMEs for situational judgment tests, enhancing interrater agreement and selection efficiency.
The development of a scoring key for the situational judgment test (SJT) often requires subject matter experts (SMEs) to identify the best responses for a hypothetical situation. And yet, there is no gold standard for identifying the SMEs. This paper describes an empirical and context-free approach: the Cochran-Weiss-Shanteau (CWS) method, which does not rely on external criteria such as tenure or credential. We first describe the theory behind the empirical approach of expertise. We also outline the CWS method and provide an R script for calculating the CWS-index. Next, we demonstrate how the CWS-index can be used for improving interrater agreement and the efficiency of SME selection. Finally, we examined the nomological network of the CWS index. We found the CWS index was associated with reflective thinking and intuition avoidance.
No takes yet. Share an insight, caveat, or question.
Zhang et al. (2021) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: