High Accuracy with Low Costs: The Pretrain-Finetune Paradigm for Classification with Transformer-based Language Models

Political science research increasingly relies on text classification to gauge subtle concepts like toxicity or anger. However, common methods treat words and phrases in isolation, ignoring the relational dynamics where meaning is embedded in the interaction between phrases and their textual environment. Recent transformer-based language models overcome such limitations, but are rarely used in published political science articles, possibly due to misconceptions about their data and computational requirements. This article explains the pretrain-finetune paradigm for transformer-based language models and its potential to improve text analysis in political science by harnessing robust models trained on extensive datasets while requiring relatively small amounts of labeled data and computational resources from researchers. To demonstrate its effectiveness, I employ the approach to identify toxic language in conversation threads following US Senators' tweets.