Beyond Agreement: Using LLM Reasoning to Evaluate Speech Acts and Politeness in Social Media

This study evaluates the ability of a Large Language Model (LLM) to analyze discourse-pragmatic phenomena in English-language social media threads by going beyond simple agreement with human-assigned annotations to incorporate the LLM’s reasoning. Content analysis methods are employed to compare Gemini 2.5 Pro and human annotators along multiple dimensions with respect to their annotation of two Reddit threads for speech acts and politeness using Computer-Mediated Discourse Analysis (CMDA) methods. Taking its reasoning into account reveals Gemini to be performing these tasks at a higher level than exact human-LLM agreement metrics do; it shows the LLM making few actual errors of either code assignment or reasoning, except for over-coding politeness. We argue that LLM reasoning, as an emerging genre of AI-mediated communication, is both a source of analyticn insight and a promising object for discourse analysis.