Materials Dual-Source Knowledge Retrieval-Augmented Generation for Local Large Language Models in Photocatalysts

Large language models (LLMs) have the potential to serve as collaborative assistants in scientific research. However, adapting them to specialized domains is difficult because it requires the integration of domain-specific knowledge. We propose Materials Dual-Source Knowledge Retrieval-Augmented Generation (MDSK-RAG), a retrieval-augmented generation (RAG) framework that enables domain specialization of LLMs for materials development under fully offline (no-Internet) operation to ensure data confidentiality. The framework unifies two complementary knowledge sources, experimental CSV data (practical knowledge) and scientific PDF literature (theoretical insights), by converting tabular records into template-based text, retrieving relevant passages from each source, summarizing them with a local LLM, and merging the summaries with the user query prior to generation. As a case study, we applied the framework to metal-sulfide photocatalysts using 740 in-house experimental records and 20 scientific PDFs. We evaluated the framework on a benchmark consisting of 14 expert-defined questions and used two-sided Wilcoxon signed-rank tests for paired comparisons. Models with fewer than 10 billion parameters were executed on a laptop, whereas larger models were run on a dedicated local server; the cloud-based LLM (GPT-4o) was evaluated via the cloud service. For practical deployment, gemma-2-9b-it (W = 14.0, two-sided p-value: p = 1.34 × 10-2) and improved the median expert 5-point rating from 2 to 3, an absolute increase of 1 point (corresponding to a relative percentage gain of 50.00%; Wilcoxon signed-rank test statistic: W = 3.5, two-sided p-value: p = 7.00 × 10-3). For reasoning-type questions, incomplete context retrieved by MDSK-RAG sometimes disrupted the model's reasoning process and led to incorrect conclusions, indicating remaining room for improvement. Comparable, statistically significant improvements were observed for the other local models (Qwen2.5-7B-Instruct and a larger gemma-2-27b-it) between conditions with and without the framework in the evaluation by cosine similarity to expert reference answers. In comparison to a cloud-based LLM, the gemma-2-9b-it with the framework outperformed GPT-4o. In this case study, the framework effectively incorporated practical experimental knowledge and theoretical literature into local LLM responses, improving accuracy for domain-specific queries. The framework presented here offers a practical and extensible adaptation of local LLMs to domain-specific scientific research.