Predicting and Evaluating Item Responses Using Machine Learning, Text‐Embeddings, and LLMs

This is the authors' abstract. We don't add key points for this paper.

This study evaluates the extent to which synthetic response generation methods reproduce the psychometric properties of Likert‐type social‐emotional assessment data. We compare four approaches: machine learning (ML), text‐embedding models, zero‐shot large language models (LLMs), and fine‐tuned LLMs, in generating responses to 10 field‐test items from the Devereux Student Strengths Assessment (DESSA). Synthetic responses were evaluated using prediction accuracy, graded response model (GRM) parameter recovery, response‐style indices, and person‐level response patterns. At an aggregate level, ML and text‐embedding approaches achieved higher accuracy and more stable recovery of threshold parameters than LLM‐based methods. However, closer examination revealed systematic distortions across all methods, including shifts in scale‐use behavior and reduced variability in person‐level response patterns, particularly for LLM‐generated data. These findings suggest that while synthetic methods can approximate surface‐level response distributions, they do not consistently reproduce the response processes underlying ordinal data. Implications for the use of synthetic data in early‐stage item development and psychometric evaluation are discussed.