AI-Assisted Sentiment Analysis of Student Voice: A Human-Validated Comparison of Lexicon and Transformer-Based Tools

Open-ended survey responses offer rich insights into the student experience, yet analyzing this qualitative data at scale is often time-consuming and labor-intensive. This study evaluates whether off-the-shelf sentiment analysis tools can validly classify such data for institutional assessment. Analyzing 1,220 free-text responses from undergraduate students at a private liberal arts university, we compared the performance of four tools: two lexicon-based (TextBlob and VADER) and two transformer-based (BERT-nlptown and RoBERTa-cardiffnlp) against manual coding. Evaluation metrics included accuracy, macro F1 score, and Cohen’s kappa (κ). While BERT-nlptown performed best, it achieved only moderate agreement with human coding (κ = .42). Furthermore, lexicon-based approaches misclassified roughly 20% of responses due to limited vocabulary coverage. Notably, all tools struggled to accurately classify negative, neutral and mixed sentiments. We recommend conducting lexicon-coverage diagnostics, few-shot prompting, domain-specific finetuning, and human-in-the-loop oversight to ensure valid, reliable, and trustworthy institutional assessment.