Empirical Evaluation of Automatic Speech Act Classification: From Logistic Regression to GPT-4o

Speech act classification aims to assign communicative functions to utterances, a task that is central to understanding pragmatic aspects of language. Automating this process can substantially reduce the need for manual annotation, enabling large-scale pragmatic analysis. In this study, we evaluate different automatic classification approaches on the SPICE Ireland dataset. This dataset has been pragmatically annotated according to Searle’s taxonomy of illocutionary acts. We compare traditional machine learning models (, , and models), a fine-tuned RoBERTa language model, and GPT-4o using zero- and few-shot prompting. Finally, by fine-tuning GPT-4o, we achieve the best performance with an accuracy of 87% and a macro average F1-score of 66%. A qualitative analysis of disagreement cases further indicates that fine-tuned GPT-4o often produces predictions that align with expert judgments, suggesting that some disagreements between model predictions and reference labels may reflect annotation ambiguity or context sensitivity rather than clear model errors.