Efficiency Analysis of Fine-Tuning Compact Language Models for Biomedical Information Retrieval Systems

The rise of semantic search systems and Retrieval-Augmented Generation (RAG) architectures in the biomedical domain poses critical challenges regarding computational costs and data sovereignty. This paper investigates the effectiveness of specializing ultra-compact language models for clinical embedding generation, utilizing the EmbeddingGemma 300M as the base model. By applying Low-Rank Adaptation (LoRA) techniques and low-level optimizations via the Unsloth library, the study demonstrates the feasibility of performing fine-tuning on entry-level hardware (NVIDIA Tesla T4). Experimental results show a 79.5% reduction in Multiple Negatives Ranking Loss (MNRL) in only 44.7 minutes, consuming just 6.56 GB of VRAM. The achieved throughput of 3,809 tokens/s and the residual operational cost (~$0.26) validate the proposal as a sustainable and private alternative. The results demonstrate competitive performance, approaching the effectiveness of significantly larger models in specific biomedical retrieval tasks with a fraction of the computational cost, thus promoting technological democratization for healthcare institutions with limited resources.