Enhancing language models with selective masking for thematic and misinformation classification in a One Health context

• Two-phase pipeline for domain adaptation and fine-tuning of language models. • Fine-tuning for classification tasks using a small amount of labeled data. • Novel selective masking strategies: SM-Lex-TFIDF and SM-NonLex-TFIDF. • Improved performance in thematic and misinformation classification in One Health. • A summary mapping each strategy to its most effective context. The objective of this paper is to address the scarcity of labeled textual data and improve the performance of language models in classification tasks within a One Health context by using small domain-specific labeled corpora. To address these challenges, we propose a two-phase training pipeline for language models, in which the first phase involves post-training guided by selective masking (SM) strategies to adapt the model to a specific domain. For this purpose, we propose two novel masking strategies: SM-Lex-TFIDF, which masks domain lexicon terms with high TF-IDF (term frequency-inverse document frequency) values, and SM-NonLex-TFIDF, which masks non-domain lexicon terms with high TF-IDF values. The second phase focuses on fine-tuning the model for the target classification task using small amounts of labeled data. To demonstrate the effectiveness of our approach, we focus on two related application areas within the One Health context, i.e., (i) thematic content in integrated health, covering the biomedical, plant health, and syndromic surveillance domains, and (ii) epidemic misinformation, to achieve improved One Health monitoring. We conduct extensive evaluations to assess the performance of our approach using three language models: BERT Base , SciBERT, and BioBERT. Additionally, we compare our method with low-resource LLM-based approaches, including zero/few-shot classification. Experimental results demonstrate significant improvements in the performance of the language models across classification tasks in both targeted areas, even with limited labeled data. Our approach outperforms zero/few-shot classification using LLaMA-3.1-8B and Mistral-7B in four out of the five datasets evaluated. Furthermore, we provide a summary mapping each strategy to its most effective context.