Beyond the classroom sample: Validating a STEM biographical data measure with online adult learners

Higher education research in STEM is a critical and evolving domain. However, it often relies heavily on college student samples, particularly those who have already signaled STEM interest via their majors or extracurricular activities, limiting the generalizability of findings. New digital learning platform based research infrastructures offer advantages that complement traditional recruitment and data collection approaches. We utilized OpenStax Kinetic to validate a biographical data (or biodata) measure of STEM and other early academic life experiences, recruiting online adult learners with no specific predisposition toward STEM. This sample spans broader interests and a wider age range than traditional college-aged STEM students. As with many measure validation studies, our current instrument is relatively long, raising concerns about participant fatigue and data quality. Kinetic enables administration of convergent and discriminant validity measures (vocational interest, STEM interest, and personality traits) at different times, reducing common-method variance and respondent fatigue. We conducted an exploratory factor analysis of a seven-factor model with online adult learners ( n = 467), finding that five factors (general parental support, STEM parental support, teacher support, math extracurricular engagement, and science extracurricular engagement) loaded as expected, while the remaining two were reconceptualized into factors comprising items that capture community STEM engagement and intrinsic hands-on engagement with STEM. We describe our interpretability assessment process and item refinement decisions for future validation efforts. Finally, we highlight additional advantages and considerations for researchers and learning scientists using large-scale platforms for measurement studies.