ContrastSkill: Task-Oriented Contrastive Pre-Training for Enhanced Skill Extraction in Job Data

Abstract The continuously evolving nature of labour markets demands a more flexible and adaptable workforce. To meet these demands, job seekers must understand competence requirements not just for specific jobs but across entire sectors and industries. While prior work has explored Named Entity Recognition (NER) and rule-based methods for skill extraction, these approaches often struggle with generalisation across diverse datasets and industries and are susceptible to errors concerning rarer competencies. This gap poses challenges for policymakers, career researchers and HR professionals who rely on accurate, large-scale skill extraction to analyse workforce trends and inform policy decisions. In this work, we introduce ContrastSkill, a contrastive learning-based framework for pre-training language models to enhance skill extraction. By adding a supervised contrastive pre-training step utilising domain data, we improve generalisation and robustness in standard transfer learning NER pipelines. In cross-dataset experiments, ContrastSkill achieves up to 2.32 percentage points span-F1 improvement over standard fine-tuning and delivers significant gains on two of three evaluated datasets, with comparable performance on the third. We also compare ContrastSkill with baseline methods, conduct a comprehensive study across different models, and perform an extensive ablation to reveal interpretability insights and optimal architectural choices for job-related skill extraction. We release the code and supplementary materials to foster reproducibility.