Establishing interobserver reliability is a key step for asserting concordance in personality ratings. Yet, typical measures of reliability-such as intraclass correlation coefficients (ICCs)-are not just reflective of whether raters agree on subject variation. Rather, ICC estimates can be influenced by a multitude of data qualities such that high reliability need not demonstrate validity. I explored what our expectations are for ICCs based on the state of published studies' estimates and based on simulations built upon the typical properties of Likert ratings used for nonhuman assessments. I relied on the nonhuman primate personality literature, assembling ICC(3, 1) estimates from 34 publications. I conducted Bayesian meta-analyses, which demonstrate that ICC(3, 1) estimates showed a posterior estimate of 0.31, but that items exhibited high variability. This variability showed concordance with the underlying traits and dimensions, as well as taxonomic distinctions. Simulations, however, demonstrated the complexities of using ICC estimates as a measure of reliability. Furthermore, the simulations showed that neat threshold cutoffs are of poor utility without consideration of the properties of the data and system, namely, the number of raters and subjects, as well as distributions of the item scores. I provide recommendations based on expectations of these data. Nonhuman primate ICC estimates show reasonable values relative to expectations of reasonable true subject variation, in line with repeatability estimations from behavior, physiology, and expectations from human individual differences. (PsycInfo Database Record (c) 2026 APA, all rights reserved).
Alexander J. Pritchard (Thu,) studied this question.