Building justice-informed validity arguments in organizational psychology: Where the evidence is (and isn't)

Introduction This review examined what types of validity evidence are presented in recent I-O psychology validation articles and the extent to which these studies incorporate justice-oriented features. Methods We reviewed 63 empirical articles published between 2021 and 2025 in five leading journals that presented new work-relevant instruments or provided validation evidence for existing ones. We used a theory-guided coding protocol to map reported evidence onto the Standards for Educational and Psychological Testing, Kane's interpretation/use argument, and Randall's justice-oriented validity framework. Results Validation evidence mostly consisted of psychometric analyses that supported domain description, scoring, generalization, and extrapolation inferences. Response-process evidence was uncommon and typically limited to evaluations of item clarity. There was limited attention to the consequences arising from score use. Similarly, justice-relevant evidence focused on statistical comparability across groups, with less emphasis to interest holder involvement, construct representation, structural inequality, or potential harm. Discussion These patterns suggest that, in the reviewed articles, psychometric adequacy was prioritized over arguments connecting scores to their meaning and use. We discuss the implications of these findings and outline the challenges of AI-enabled assessments. Stronger validity arguments will need clearer specification of the intended uses of score, greater involvement of affected interest holders, increased focus on respondent interpretation and cultural context, consideration of justice and equity throughout the validation process, and evaluation of the consequences of score use.