DroPacter : Compacter-based Tuning with Layer Freezing in Pre-Trained Language Model

Obtaining high-quality labeled dataset for training a cybersecurity language model is challenging due to sensitive content, proprietary information, and a shortage of skilled annotators. To address this issue, we turn to pre-trained language models (PLMs) but fine-tuning them can be costly in terms of time, storage, and labeled data as model complexity grows. Previous research introduced lightweight adapter modules in which only a small set of parameters are updated for each new task during model training. In our work, we concentrate on enhancing Compacters (Compact Adapters) and present DroPacter, a unique fusion of compacters and layer freezing strategies. We apply DroPacter to a state-of-art cybersecurity PLM and evaluate it on entity extraction downstream task. Our preliminary experiments show that DroPacter achieves accuracy on-par with full model fine-tuning with 50% of less trainable parameters.