Unpacking the grammar of psychological tests: Item response modeling and corpus analysis as a new lens on validity

Complex syntax in psychological test items may hinder some respondents’ comprehension, yet the prevalence of specific syntactic complexity features and their implications for item responding and validity are not well understood. This two-part study provides new empirical insight into these relationships. Study 1 examines whether different types of dependent clauses are associated with item responses and whether these associations vary according to respondents’ vocabulary familiarity. Study 2 uses a corpus-based register analysis to determine how frequently dependent clause types occur in psychological test items, providing context for the practical significance of the Study 1 findings. Of the five clause types examined in Study 1, only nonfinite complement clauses showed a significant association with item responses, and this association was moderated by vocabulary familiarity. Study 2 further showed that nonfinite complement clauses are the most prevalent dependent clause type in a corpus of psychological test items. Together, these findings suggest that dependent clause types should not be treated as uniformly difficult to process. This study demonstrates how integrating psychometric modeling with corpus-based linguistic analysis can build validity evidence and inform fairer assessment practices for linguistically diverse populations.