Item Selection as the Cause and Solution to Validity Problems in Psychology

AbstractThe validity of psychological measures depends upon selection of items—a process often handledsubjectively and informally. We highlight problems resulting from neglected aspects of indicatorsampling and investigate how item selection can help to resolve pervasive validity issues (e.g., poorcontent coverage, jingle-jangle fallacies). We discuss three stages of item selection: Stage 1, definingthe item universe, which requires a clear, non-overlapping delineation of psychological constructs; 2)creating an initial item pool through comprehensive approaches such as cumulative item pools toensure full content coverage; and 3) compiling final scales using psychometric optimizationprocedures such as Ant Colony Optimization (ACO). We argue that traditional prioritization of highinternal consistency in Stage 3 leads to overly redundant and conceptually narrow scales and proposeexpanded selection strategies aimed at maximizing content coverage and optimizing the nomologicalnet (convergent/discriminant validity). Overall, rigorous consideration of item selection is crucial forsignificantly enhancing the generalizability and validity of psychological assessment. We advocate forviewing item sampling as a strategic intervention for validity enhancement rather than a passive,ritualistic filtering step.