Game-Based Assessments (GBAs) offer innovative approaches to measuring psychological constructs by embedding tasks within interactive environments. Despite their potential, concerns remain about validity and whether gameplay reflects stable traits rather than situational or design-driven factors. This study aimed to examine the construct validity of a GBA designed to measure Conscientiousness by integrating cognitive interviewing, Item Response Theory, and design evaluation. Ten participants completed the GBA and participated in follow-up interviews guided by Collins’ framework. The study addressed how gameplay tasks represented Conscientiousness and which design features affected players’ cognitive responses. Findings showed that early stages (1–2) demonstrated strong alignment, clear instructions, understandable goals, and items with adequate fit and discrimination, eliciting structured, trait-consistent behaviors. Later stages (3–6) exhibited weaker psychometric performance, divided attention, and confusion from visual similarity. Overall, results highlight the need to align psychometric rigor, cognitive clarity, and design simplicity in developing valid GBAs.
Fadillah et al. (Fri,) studied this question.