Text Psychometrics: Assessing psychological constructs in text using natural language processing

Large language models (LLMs) have revolutionized natural language processing (NLP). Yet when used to assess psychological constructs in text, they are generally not evaluated for the types of validity, reliability, and standardization typically expected from traditional questionnaires with rating scales. This study bridges that gap by proposing a framework for evaluating the psychometric properties of NLP-based psychological assessments (including LLMs-based ones), an area of research we refer to as Text Psychometrics. We first review different NLP methods, compare their ability to address key challenges in psychological research such as explainability, and outline ways of evaluating them on many desirable psychometric properties. Our goal is to describe most psychometric properties that may be of interest, especially for psychology and NLP researchers who are less familiar with psychometrics or psychometricians who are less familiar with NLP. We then illustrate the tradeoffs of different NLP methods through two empirical studies. Study 1 classifies thousands of crisis counseling conversations and Reddit posts into different types of mental health issues and introduces a novel method to evaluate text models for content validity—the extent to which a test captures the full range of expressions of a single construct. Study 2 examines prospective criterion validity by estimating how 49 known suicide risk factors predict imminent risk in crisis counseling conversations. We find that mentioning lethal means for suicide, the police, depressed mood, and anxiety in text as well as the texter writing more than the counselor characterize the conversations that end up needing an emergency service intervention for imminent suicide risk, which showcases the interpretability of the method that was used. In sum, NLP studies in psychology often rely on only a few validation metrics; here, we demonstrate the need for broader psychometric evaluation and, as a step towards Text Psychometrics, provide practical examples and future directions that can help researchers evaluate these models.