What question did this study set out to answer?

This research aims to evaluate the effectiveness of LoRA and Prompt Tuning for fine-tuning language models in low-resource conditions.

June 2, 2026Open Access

Parameter-Efficient Fine-Tuning of Small Pre-Trained Language Models Using LoRA and Prompt Tuning: A Low-Resource Evaluation on DistilBERT

Key Points

This research aims to evaluate the effectiveness of LoRA and Prompt Tuning for fine-tuning language models in low-resource conditions.
Controlled experimental setup comparing LoRA and Prompt Tuning for binary sentiment classification with DistilBERT.
Training data size of 1,000 samples used for main comparisons; also evaluated with 100 samples.
Metrics evaluated include accuracy and the number of trainable parameters for both methods.
LoRA achieved an accuracy of 88.7% with 1,000 samples, while Prompt Tuning achieved 82.5% with 15,360 trainable parameters.
LoRA maintained 70.0% accuracy even with only 100 training samples.
Cross-domain results showed LoRA at 86.8% and Prompt Tuning at 84.4% accuracy on Yelp review data.

Abstract

The process of adapting large pre-trained language models (lms) to tasks with few-labeled data remains a major challenge because of the restrictions on data and computation. This paper mainly discusses two parameter-efficient methods of fine-tuning, Low Rank Adaptation (LoRA) and Prompt Tuning, for binary sentiment classification where the base model is distilbert in low-resource situations. In order to make a fair comparison of the two methods, a controlled experimental setup is used.Experiments have shown that with the size of training data 1,000 LoRA can produce the result 88.7% in terms of accuracy, whereas Prompt Tuning can produce 82.5% but with far fewer trainable parameters (15,360). Even with extremely low-resource conditions (100 samples), LoRA still can have a performance of 70.0%. The results for cross-domain on Yelp review data show that LoRA obtains 86.8% accuracy, whereas Prompt Tuning 84.4%, which is also indicative of a good generalization ability. The result of this work is that LoRA is shown to be a better choice if the performance within the domain is desired, while Prompt Tuning is a better choice in the case of limited parameter budget but will have comparable performance in transfer task. Besides, they shed light on the practical knowledge of choosing fine-tuning methods in resource-constrained environments.

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper