Generative Psychometrics via AI-GENIE: Automatic Item Generation and Validation via Network-Integrated Evaluation

The rapid advancement of artificial intelligence (AI), particularly large language models (LLMs), has introduced powerful tools for various research domains, including psychological scale development. This study presents a fully automated methodology for efficiently generating and selecting high-quality, non-redundant items for psychological assessments using LLMs and network psychometrics. Our approach called, Automatic Item Generation and Validation via Network-Integrated Evaluation (AI-GENIE), reduces reliance on expert intervention by integrating generative AI with the latest network psychometric techniques. The efficacy of AI-GENIE was evaluated through Monte Carlo simulations using the Mixtral, Gemma 2, Llama 3, GPT 3.5, and GPT 4o models to generate item pools that mimic Big Five personality assessment. The results demonstrated a significant improvement in item selection efficiency, with increases of 2.3-15.4% in normalized mutual information in the final item pool across all models. These findings suggest that AI-GENIE is a highly effective tool for automating and streamlining scale development and validation process.