An Analytic Developmental Rubric for AI-Enabled Integrated STEM Instructional Design: Development and Validation with Preservice Teachers

Teachers’ artificial intelligence (AI) competence is increasingly important in China’s educational modernization. Yet in contexts related to science, technology, engineering, and mathematics (STEM), how AI integration at the pedagogical level can be made observable in instructional design remains underexamined. This mixed-methods study developed an analytic developmental rubric that integrates the AI pedagogy aspect of UNESCO’s AI Competency Framework for Teachers with seven key characteristics of integrated STEM. The rubric uses gates to define the minimum evidence required for each STEM characteristic and uses verifiable descriptors and hallmarks to assign a competency level within each characteristic for preservice teachers’ use of AI to enable integrated STEM instructional design. Then, the rubric underwent a series of validation procedures. Eight experts from two universities conducted expert review and content validity assessment. A total of 108 preservice teachers from six teacher education degree programs spanning mathematics, science, and technology at a normal university in China completed integrated STEM instructional designs on the same topic. Three raters from the two universities independently scored the artifacts, and the resulting scores were used to estimate inter-rater reliability, examine intercorrelations and internal consistency, and assess known-group validity. The findings provided evidence supporting the rubric’s reliability and validity and indicated that it could consistently distinguish preservice teachers across training stages ordered a priori by expected competency. In this way, this rubric begins to fill the gap in the assessment of AI-enabled integrated STEM instructional design, and its scope of application is not confined to any particular STEM subject or instructional design framework.