Exploring the Effect of Human Error When Using Expert Judgments to Train an Automated Scoring System

Educational Measurement: Issues and PracticeEarly View Original Article Exploring the Effect of Human Error When Using Expert Judgments to Train an Automated Scoring System Stephanie Iaccarino, Stephanie Iaccarino National Board of Medical ExaminersSearch for more papers by this authorBrian E. Clauser, Brian E. Clauser National Board of Medical ExaminersSearch for more papers by this authorPolina Harik, Polina Harik National Board of Medical ExaminersSearch for more papers by this authorPeter Baldwin, Peter Baldwin National Board of Medical ExaminersSearch for more papers by this authorYiyun Zhou, Yiyun Zhou National Board of Medical ExaminersSearch for more papers by this authorMichael T. Kane, Michael T. KaneSearch for more papers by this author Stephanie Iaccarino, Stephanie Iaccarino National Board of Medical ExaminersSearch for more papers by this authorBrian E. Clauser, Brian E. Clauser National Board of Medical ExaminersSearch for more papers by this authorPolina Harik, Polina Harik National Board of Medical ExaminersSearch for more papers by this authorPeter Baldwin, Peter Baldwin National Board of Medical ExaminersSearch for more papers by this authorYiyun Zhou, Yiyun Zhou National Board of Medical ExaminersSearch for more papers by this authorMichael T. Kane, Michael T. KaneSearch for more papers by this author First published: 07 August 2025 https://doi.org/10.1111/emip.70002Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookxLinkedInRedditWechatBluesky References Attali, Y., Powers, D., Freedman, M., Harrison, M., Obetz, S. (2008). Automated Scoring of Short-Answer Open-Ended GRE® Subject Test Items. GRE Board Research Report No. 04-02 ETS RR-08-20. ETS, Princeton, NJ Google Scholar Baker, E. L., O'Neil, H. F., Jr., & Linn, R. L. (1993). Policy and validity prospects for performance-based assessment. American Psychologist, 48, pp. 1210–1218. 10.1037/0003-066X.48.12.1210 Web of Science® Google Scholar Bennett, R. E., & Ward, W. C. (1993). Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. Google Scholar Bexte, M., Horbach, A., & Zesch, T. (2022). Similarity-based content scoring-how to make S-BERT keep up with BERT. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022) (pp. 118–123). Association for Computational Linguistics. Google Scholar Braun, H. I., Bennett, R. E., Frye, D., & Soloway, E. (1990). Scoring Constructed Responses Using Expert Systems. Journal of Educational Measurement, 27, pp. 93–108. 10.1111/j.1745-3984.1990.tb00736.x Web of Science® Google Scholar Brennan, R. L. (2001b). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38, pp. 295–317. 10.1111/j.1745-3984.2001.tb01129.x Web of Science® Google Scholar Brennan, R. L. (2001a). Generalizability theory. Springer-Verlag. https://doi.org/10.1007/978-1-4757-3456-0 10.1007/978-1-4757-3456-0 Google Scholar Clauser, B. E., Yaneva, Y., Baldwin, P., Ha, L. A. & Mee, J. (2024). Automated scoring of short-answer questions: A progress report. Applied Measurement in Education, 37(3), pp. 209–224. https://doi.org/10.1080/08957347.2024.2386945 10.1080/08957347.2024.2386945 Web of Science® Google Scholar Clauser, B. E., Harik, P., & Clyman, S. G. (2000). The generalizability of scores for a performance assessment scored with a computer-automated scoring system. Journal of Educational Measurement, 37, pp. 245–262. 10.1111/j.1745-3984.2000.tb01085.x Web of Science® Google Scholar Edgeworth, F. Y. (1888). The statistics of examinations., Journal of the Royal Statistical Society, 51, pp. 346–368. 10.1111/j.2397-2335.1888.tb00076.x Google Scholar Fiske, E. B. (1990). But is the child learning? Schools trying new tests. The New York Times, Al, B6. Google Scholar Frederiksen, J. R., & Collins, A. (1989). A systems approach to educational testing. Educational Researcher, 18(9), pp. 27–32. https://doi.org/10.2307/1176716 10.3102/0013189X018009027 Google Scholar Gulliksen, H. (1936). The content reliability of a test. Psychometrika, 1, pp. 186–194. 10.1007/BF02288365 Google Scholar Guthrie, J. T. (1984). Testing higher level skills. Journal of Reading, 28(2), pp. 188–190. http://www.jstor.org/stable/40029442 Web of Science® Google Scholar Harik, P., Clauser, B. E., Murray, C., Artman, C., Veneziano, A., Margolis, M. (2013). Comparison of automated scores derived from independent groups of content experts. San Francisco, CA: National Council on Measurement in Education (NCME) 2013 Annual Meeting. Google Scholar Kane, M. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement ( 4th ed., pp. 17–64). American Council on Education; Praeger. 10.3917/rhu.016.0017 Google Scholar Kelly, F. J. (1915). The Kansas Silent Reading Test. Topeka, KS: Kansas State Printing Plant. Google Scholar Mee, J., Pandian, R., Wolczynski, J. Morales, A., Paniagua, M., Harik, P., Baldwin, P., Clauser, B. E. (2024). An experimental comparison of multiple-choice and short-answer questions on a high-stakes test for medical students. Advances in Health Science Education, 29, pp. 783–801. https://doi.org/10.1007/s10459-023-10266-3 10.1007/s10459-023-10266-3 PubMed Web of Science® Google Scholar Nickerson, R. S. (1989). New directions in educational assessment. Educational Researcher, 18(9), pp. 3–7. https://doi.org/10.2307/1176712 10.3102/0013189X018009003 Google Scholar Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. arXiv preprint arXiv:1908.10084. Google Scholar Sam, A. H., Hameed, S., Harris, J., & Meeran, K. (2016). Validity of very short answer versus single best answer questions for undergraduate assessment. BMC Medical Education, 16(1), pp. 1–4. https://doi.org/10.1186/s12909-016-0793-z 10.1186/s12909-016-0793-z PubMed Google Scholar Suen, K. Y., Yaneva, V., Ha, L. A., Mee, J., Zhou, Y., Harik, P. (2023) ACTA: Short-answer grading in high-stakes medical exams. To appear in Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Toronto, Canada, July 14, 2023. Google Scholar Early ViewOnline Version of Record before inclusion in an issue ReferencesRelatedInformation