Lessons Learned About Transparency, Fairness, and Explainability from Two Automated Scoring Challenges

The National Assessment of Educational Progress (NAEP) conducted two automated scoring challenges with fourth and eighth grade reading (n=20 items, 460k responses) and math (n=10, 260k responses) constructed response items to evaluate the accuracy of AI scoring and to effective methods that could be responsibly included in operational scoring. Submissions required technical reports explaining methods used, interpreting modeling results and conducting fairness analysis. Results demonstrated that automated scoring was nearly as accurate as human raters, despite extremely high levels of baseline human scoring accuracy (Math ~.95 QWK, Reading ~.905 QWK). Biased results were encountered on some items by ELL and IEP status