Measure Length in Research Settings: Effects on Cognitive and Non-Cognitive Scores and Implications for Psychometric Properties

When administering measures, researchers are often concerned about the psychological burden of measure length (i.e., the number of items) on participants and its potential effect on test scores and resulting psychometric properties. The present work experimentally examined this issue in two studies using two independent samples of undergraduate students and a between-subjects design whereby participants of each sample were randomly assigned to complete either a short or long measure. Study 1 (N = 398) examined the effect of measure length on careless responding on a non-cognitive measure and the resulting psychometric properties (i.e., internal consistency and construct validity). Study 2 (N = 426) examined the effect of measure length on performance scores on a cognitive measure and the same resulting psychometric properties. Measure length had no effect on performance on the cognitive measure, and for the non-cognitive measure had an inconsistent effect on careless responding. Pertaining to psychometric properties, measure length had no meaningful effect on the internal consistency of both measures. The effect on construct validity depended on the criterion, but on average, was small to negligible. Overall, the results provide little evidence to support researchers’ concerns about the potential adverse psychological effects associated with using longer measures.