Do all politicians sound the same? Comparing model explanations to human responses

It is sometimes said that all politicians sound the same with their speeches mired in political jargon full of clichés and false promises. To investigate how distinct the plenary speeches of political parties truly are and what linguistic features make them distinct, we trained a BERT classifier to predict the party affiliation of Finnish members of parliament from their plenary speeches. We contrasted and compared model performance to human responses to see how humans and the model differ in their ability to distinguish between the parties. We used the model explainability method SHAP to identify the linguistic cues that the model most relies on. We show that a deep learning model can distinguish between parties much more accurately than the respondents to the questionnaire. The SHAP explanations and questionnaire responses reveal that whereas humans tend to rely on mostly topical cues, the model has learned to recognize other cues as well, such as personal style and rhetoric.