Beyond the Numbers: Using Large Language Models to Analyze Student Feedback in Large Psychology Courses

Background: Psychology instructors struggle to analyze qualitative student feedback in large courses where traditional Likert-scale evaluations fail to capture student experience complexity. Current approaches are either too time-consuming or lack contextual understanding for actionable insights. Objective: To develop and validate a framework using open-source Large Language Models (LLMs) to analyze student feedback in psychology courses, comparing LLM insights with traditional evaluation metrics. Method: We implemented Facebook's BART LLM using zero-shot classification on open-ended course evaluations from a large psychology course comparing traditional lecture with active learning formats across two semesters. Data included 270 evaluations yielding 678 responses, analyzed using four learning categories. Results: LLM analysis revealed striking discrepancies with traditional metrics. While Likert-scale responses showed minimal differences between formats (Cohen's d = 0.16-0.33), LLM analysis revealed large, significant effects across all dimensions (Cohen's d = 0.94-1.13, p .001). Validation confirmed reliability through moderate correlations with related Likert items (r = 0.29-0.44). Conclusion: LLM analysis demonstrated superior sensitivity in detecting teaching approach differences, capturing qualitative distinctions that numerical ratings miss while addressing challenges of qualitative data volume and analysis time. Teaching Implications: This methodology enables instructors to analyze hundreds of responses in minutes using accessible tools, providing practical evidence-based teaching improvement insights.