Low-Resource Fine-Tuning of LLMs for Domain-Specific Tasks

Large Language Models (LLMs) such as GPT and LLaMA have demonstrated remarkable capabilities across diverse natural language processing (NLP) applications. However, their enormous computational and memory requirements hinder adoption by smaller research labs and enterprises. Full-scale fine-tuning of such models is often infeasible due to high GPU memory, storage, and energy consumption. Parameter-Efficient Fine-Tuning (PEFT) techniques, including Low-Rank Adaptation (LoRA), adapter-based methods, and prefix-tuning, present an alternative for adapting LLMs to downstream tasks under constrained budgets. Despite progress in PEFT for general NLP benchmarks, limited attention has been given to domain-specific applications such as healthcare and energy, where specialized knowledge is critical. This research investigates low-resource fine-tuning strategies for domain adaptation of LLMs, identifies the challenges of constrained environments, and evaluates practical frameworks that balance efficiency and performance. Experimental results demonstrate that LoRA- and adapter-based methods achieve competitive accuracy while drastically reducing trainable parameters and compute costs, making them highly suitable for resource-limited settings.