System-Level Efficiency of LLM Fine-Tuning Strategies in Resource-Constrained Environments: A Comparative Study of LoRA, Full Tuning, and Training from Scratch

This study examines an often-overlooked dimension of fine-tuning large language models (LLMs): the impact of different strategies on system-level resource consumption in hardware-constrained environments, particularly for deployment in settings such as academic labs, low-cost virtual machines, or edge devices. We present a comparative investigation of three approaches: (1) training from scratch with no pretrained weights, (2) full fine-tuning of all parameters, and (3) parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA). All experiments were conducted under identical VirtualBox settings-four CPU cores, 5043 MB RAM, and, when necessary to prevent system failure, 8 GB of swap space. A custom Bash script continuously logged real-time system metrics, including CPU utilization, memory footprint, and disk I/O activity. The experiment was executed in two phases: Phase 1 employed the IMDB dataset (50k reviews), while Phase 2 scaled the study to a larger dataset, the Amazon Book Reviews (100k samples), to evaluate system behavior under big data conditions. Results from Phase 1 showed that LoRA was the most resource-efficient method, completing training without reliance on swap space, whereas both scratch training and full fine-tuning required it heavily. In Phase 2, LoRA successfully completed training with swap enabled, while the other two methods failed to converge even with additional swap resources. Further analysis of the collected OS-level metrics revealed complementary insights: CPU utilization highlighted differences in computational overhead and I/O interruptions, memory usage reflected actual resource saturation and potential risks of catastrophic forgetting, and disk I/O patterns distinguished write-heavy from read-heavy workloads. Together, these findings demonstrate that system-level evaluation provides a practical lens for understanding the efficiency of fine-tuning strategies, with LoRA emerging as a robust option for constrained environments.