Fine-tuning a BERT-Base model for specific tasks, such as sentiment analysis, has become resource-intensive and often requires high computational power and memory. This paper introduces SCALE, a novel resource-efficient fine-tuning method that targets the most critical transformer layers, which reduces computational costs without sacrificing performance. By dynamically profiling transformer layers via activation magnitudes and attention entropy, SCALE selects and adapts only the most influential layers with lightweight adapter modules. The proposed method outperforms traditional fine-tuning techniques, achieving a 2.3% improvement in accuracy on the IMDB dataset and reducing training time by 56.3% compared to full-model fine-tuning. Experiments across various sentiment analysis benchmarks demonstrate SCALE’s effectiveness in optimizing fine-tuning for the BERT-base model in resource-constrained environments, achieving up to 99% of the performance of full-model fine-tuning while using only 40% of the parameters. The empirical validation in this study is restricted to binary and multi-class sentiment classification. The evaluation specifically reflects effectiveness in sentiment analysis text classification tasks.
Basahel et al. (Tue,) studied this question.