When ratings from one source have been averaged, but ratings from another source have not: Problems and solutions.

Studies using the single-aggregate approach (L. R. James, 1982), where assessments made by individual respondents are correlated with other assessments that have been averaged across multiple respondents, can exhibit a systematic bias of 20 to 70% or more if they are used to estimate individual level relationships. Not only may results of such studies be erroneous, but theory development based on such studies may be misguided. A comprehensive solution (nested and crossed designs) to the single-aggregation problem is provided through generalizability theory. Results show that the aggregation bias is a function of both the generalizability (reliability) of individual responses and the number of individuals per group. Conceptual parallels to classical measurement theory are discussed. Factors are presented for converting single-aggregated correlations and standard deviations to estimates of the corresponding values using the individual as the level of analysis.