A closer look at the impact of gender on test performance through a multivariate G-theory approach

Psychometricians have voiced their concern over the psychometric quality of a test in the realms of educational measurement and language assessment. Validity and reliability are interdependent, and sources of construct-irrelevant variance may jeopardize the validity of test score interpretations and use, leading to inconsistencies in test scores. Thus, the investigation of such irrelevant sources of variance is of utmost importance for test users. The current study used multivariate Generalizability Theory to examine whether gender was associated with differences in the dependability of scores from a university entrance examination. The outcomes revealed uneven relative contributions of the sources of variance across gender groups, alongside moderate divergences in dependability indices. Decision studies further indicated that, under an equal-items-per-subtest design, the two groups would need different numbers of items to reach a composite G-coefficient of 0.90.