Pseudo-Discrimination Parameters from Language Embeddings

As large language models are increasingly being used in psychological research, recent work hasstarted using language embeddings to generate and/ or evaluate psychometric scales. Building onthis work, we introduce and evaluate pseudo-item discrimination parameters, which approximateempirical measurement model parameters from natural language embeddings before anyresponse data are collected. Using two separate personality measures, the G50 and BFAS-100,and two independent datasets, we demonstrate initial internal and predictive validity for pseudoitemdiscrimination parameters. Results indicated that pseudo-discrimination estimates haveinternal convergent and discriminant validity and were also significant predictors of empiricaldiscrimination parameters for most personality subscales tested. Overall, pseudo-discriminationparameters estimated from language-embeddings appear to capture meaningful semanticinformation that explains variance in empirical item discrimination. In other words, pseudodiscriminationestimates provide an approximation of empirical measurement model parametersprior to collecting empirical data. Pseudo-discrimination estimates appear informative aboutactual item discrimination, easing the burden associated with item pool creation and calibration.