Is Generative Psychometrics Viable in Spanish? Evaluating an LLM-Based Automated Framework for Scale Development

This study assessed the viability of the AI-GENIE methodological framework for the construction and validation of a psychometric instrument in Spanish. Through a Monte Carlo simulation, 900 samples were generated using six large language models (LLM) and three temperature parameters, producing approximately 121,717 items that were refined using the AI-GENIE steps: Unique Variable Analysis (UVA), Exploratory Graph Analysis (EGA), and bootEGA. Following manual post-hoc refinement, the 23-item HEXACO-IA-P scale was constructed as a proof-of-concept test case, and it was administered to 252 Peruvian university students. Results demonstrated that AI-GENIE possesses an aceptable viability, successfully recovering the six-dimensional theoretical structure with a Normalized Mutual Information index (NMI) of .942 and node stability exceeding .70 for most items. Nevertheless, limitations related to semantic redundancy, variability across LLM models, and linguistic bias in algorithm training were identified. It is concluded that, although AI-assisted automated item generation constitutes a promising tool for modern psychometrics, its implementation in Spanish requires expert human supervision, methodological adjustments, and rigorous empirical validation to ensure the validity and reliability of the developed scales.