To aggregate or not to aggregate: Effect variant measurement models and their advantages for questionnaire data in experimental psychology

Experimental psychological research frequently relies on aggregating items, assuming that the effects of other variables remain constant across items. This assumption can lead to biased statistical inferences, most notably an inflated Type I error rate, as it disregards the random variability of item-specific effects. This paper highlights the potential biases that can arise from using aggregated scores. It shows the conditions under which effect-variant models, which treat both participants and items as random factors, are more appropriate than traditional aggregation approaches. We introduce the Aggregation Standard Error Inflation (ASEI), a novel metric designed to quantify the variance inflation associated with conventional aggregation methods and the related risks. Through a series of simulations and real data analyses, we show that neglecting item-specific variability in aggregated models increases Type I error rates, particularly in larger samples. The extent of this inflation can be effectively assessed using the ASEI. These findings highlight the importance of incorporating random variability across both items and participants to improve the validity of statistical inferences in psychological research that relies on questionnaires and scales.