Aligning open educational resources to new taxonomies: How AI technologies can help and in which scenarios

Aligning open educational resources (OER) to skill taxonomies is a common task in the education field and helps teachers better locate material that aligns with the standards of their curriculum. When taxonomies change, as they periodically do, re-tagging the increasing mass of open educational resources is needed. The process of manual tagging is, however, exceedingly labor intensive. We propose and evaluate a novel combination of machine learning methods to help automate tagging open educational resources with skills from an existing taxonomy as well as skills from any newly introduced taxonomy. We collected text, image figures, and videos from tens of thousands of educational resources from two major digital learning platforms to answer the research questions of: how effective are machine learning models in automatically updating OER classification to reflect a new taxonomy (RQ1), and which models may be of practical use in different scenarios (RQ2)? Using several taxonomies, including the US Common Core, we find that while full automation is not practically viable, our most generalizable model can reach non-expert human labeling performance requiring only 100 labeled examples and near expert level with 5000. We believe these novel findings may have immediate utility for practitioners and policymakers and better ready the growing landscape of open educational resources for the advent of new taxonomies ahead. We publicly release our pre-trained US Common Core and new taxonomy tagging models, providing guidance on their viability in various real-world scenarios.