The National Institutes of Health refused to switch to disaggregated ratings as a method for evaluating proposals, because no contest between disaggregated and holistic ratings had ever used scientific materials as the to‐be‐rated stimuli. We designed two studies to fill this research void. Participants rated scientific convention presentations either using a holistic procedure in which one overall rating was given or a disaggregated procedure in which one rating was given to each of five criteria. In four of the six convention sessions the disaggregated ratings led to higher inter‐rater reliability than did the holistic ratings; three of these differences were statistically significant. The inter‐rater reliabilities between the two types of ratings collapsed across all sessions differed significantly. In a second experiment, participants rated posters using either disaggregated or holistic ratings. The disaggregated ratings again led to higher inter‐rater reliability, but not significantly so. In 35 of the 43 sessions in which disaggregated and holistic ratings were compared, the variance of the disaggregated ratings was smaller.
No takes yet. Share an insight, caveat, or question.
Arkes et al. (2006) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: