Enhancing Large Language Models for Specialized Domains: A Two-Stage Framework with Parameter-Sensitive LoRA Fine-Tuning and Chain-of-Thought RAG

Large language models (LLMs) have shown impressive general-purpose language capabilities, but their application in specialized domains such as healthcare and law remains limited due to two major challenges, namely, a lack of deep domain-specific knowledge and the inability to incorporate real-time information updates. This paper focuses on addressing these challenges by introducing parameter-sensitive low-rank adaptation (LoRA) and retrieval-augmented generation (RAG), named SensiLoRA-RAG, a two-stage framework designed to enhance LLM performance in domain-specific question-answering tasks. In the first stage, we propose a parameter-sensitive LoRA fine-tuning method that efficiently adapts LLMs to specialized domains using limited high-quality professional data, enabling rapid and resource-efficient specialization. In the second stage, we develop a chain-of-thought RAG mechanism that dynamically retrieves and integrates up-to-date external knowledge, improving the model’s ability to reason with current information and complex domain context. We evaluate our framework on tasks in the medical and legal fields, demonstrating that SensiLoRA-RAG significantly improves answer accuracy, domain relevance, and adaptability compared to baseline methods.