Explaining alignment between LLM classifications of political arguments and human judgments

Large language models are increasingly used in civic tasks such as classifying political arguments, raising questions about when their classifications align with human judgments. This study examines whether LLMs can distinguish evidence-supported from rhetorical-value-based arguments and which linguistic and structural features explain their alignment with human annotations. We analyze 600 French and German arguments from Swiss referendum debates (1977–2016), manually annotated by type of justificatory support, and evaluate four models using logistic regression and linguistic features. Model–human agreement is moderate and uneven across models, and alignment is associated with broad structural features such as argument length, causality markers, and readability. Analysis of model-generated justifications shows that explanations are linguistically consistent with the assigned label and should be interpreted as post-hoc rationalizations rather than as evidence of reasoning.