Automating the quality prediction of survey questions: predicting measurement quality directly from natural text

Abstract The survey quality predictor (SQP) is a web-based tool designed to predict the measurement quality of survey questions based on up to 72 manually coded formal and linguistic characteristics (e.g. domain, response scale properties, linguistic complexity). Users must input these features following a detailed coding manual, after which a trained random forest (RF) model predicts measurement quality. Here, we evaluate whether measurement quality can instead be predicted directly from the natural language text of a survey question, eliminating the need for manual coding. We find a fine-tuned language model can predict survey item quality based solely on the question and answer options, achieving performance comparable to the RF model currently implemented in SQP, which is trained on manually coded features. Specifically, we fine-tuned xlm-RoBERTa, a multilingual transformer-based model trained on multiple text corpora in over 100 languages, using the SQP dataset. Our findings suggest the current SQP web interface (https://sqp.gesis.org), which requires users to manually code the 72 features, can be simplified. A redesigned, more user-friendly web interface could allow users to only enter the survey question and answer options, with the model automatically predicting measurement quality. This would enhance accessibility and lower the barrier for researchers and practitioners using SQP.