Inter-rater Agreement Indices for Multiple Informant Methodology

This paper addresses the potential applications of multiple informant methodology (MIM) to family research. The MIM is used when data on the same unit of analysis are collected from more than one informant, and when the researchers’ purpose requires a dyadic or group level of analysis. The MIM relies on inter-rater agreement (IRA) indices, which are needed to both estimate agreement among informants and aggregate scores from different informants. This review describes the main IRA indices: reliability within-group index (rWG), agreement within-group index (aWG), average deviation indices (ADM and ADMd), intra-class correlation coefficient (ICC), and random group index (rWG). For each index, we describe the aim, application contexts, formula(s), ranges, cutoffs, requirements to apply the index, and its possible application to family research.