The Reliability-First Myth in Item Selection – Why Item Difficulty Matters

Psychological tests are commonly evaluated using internal consistency, indexed by Cronbach's α. This emphasis fosters the assumption that maximizing α is sufficient for building useful scales – a position we call the reliability-first myth. The present study illustrates a known but practically important implication of classical test theory: reliability does not uniquely determine how well observed scores support individual-level interpretation. Using a Monte Carlo simulation within a tau-congeneric framework, we varied item-difficulty composition, test length, number of response categories, and mean factor loading. Differentiation was operationalized using the Critical Difference (CD). Response-category count, item-difficulty composition, and test length explained substantial variation in CD, whereas achieved α contributed little unique value. Extreme-difficulty sets yielded the smallest CDs but only by compressing observed-score variance. Mixed-difficulty sets cover the trait more broadly without losing precision. Reliability is therefore necessary but not sufficient for individual-level interpretation.