Key points are not available for this paper at this time.
• Explores Generalizability Theory's role in AI-driven educational research. • Addresses AI's variability in educational assessment and predictive analytics. • Proposes G-Studies and D-Studies to enhance AI-based tools' reliability. • Highlights AI-induced bias mitigation through variance decomposition. • Outlines methods to ensure equitable educational outcomes using AI. The rise of AI in education presents both transformative opportunities and methodological challenges. This paper revisits Generalizability Theory (G-Theory) as a robust framework to assess the reliability and fairness of AI-driven tools across diverse educational contexts. It is argued that G-Theory’s variance decomposition logic is uniquely suited to disentangle the multifaceted sources of error introduced by evolving AI systems, user diversity, and complex learning environments. Through empirical use cases it is illustrated how G-Theory can support the design of equitable, scalable, and context-sensitive AI applications. We further A G-Theory Readiness Checklist to guide researchers in designing studies with AI as a methodological facet is proposed. Finally, conceptual, technical, ethical, pedagogical, and regulatory limitations and implications for study designs are highlighted. The paper concludes with suggestions for future research.
Ben Degen (Thu,) studied this question.