Exploring Psychometric Analysis of Textual Data with Large Language Models: Chances and Challenges

This research explores a psychometric method for analyzing textual data using contextualembeddings generated by large language models. The proposed method leveragescontextual embeddings to create contextual scores, which are then used as input to createfactor analysis models. By treating documents as individuals and words as items, thisapproach provides a natural psychometric interpretation, unveiling a vast, unexploredlandscape for future research. It assumes that certain words, whose contextual meaningsvary significantly across documents, can be employed to differentiate documents within acorpus. The modeling process consists of two stages: obtaining contextual scores andperforming factor analysis. The first stage involves utilizing natural language processingtechniques and encoder-based transformer models to find common keywords and generatecontextual scores. The second stage involves employing different types of factor analysis toextract and define factors, obtain factor correlations, and identify top words of the factors.Tested on the Wiki STEM corpus, the experimental results demonstrate the new method’schances and challenges for analyzing textual data by combining both large language modelsand psychometric modeling.