Implementation and Performance Comparison Study of a Retrieval-Augmented Generation (RAG)-based Chatbot for Korean Medical Consultation (Preprint)

This is the authors' abstract. We don't add key points for this paper.

BACKGROUND This study focuses on the development of a GPT-4-based chatbot specifically designed for health consultations in Korean. The chatbot is implemented using the Retrieval-Augmented Generation (RAG) technique alongside Metadata Filtering to enhance its performance. OBJECTIVE The objective of this research is to analyze and compare the performance of the RAG-based chatbot against other leading language models in the context of Korean health consultations. METHODS We developed a Korean medical consultation system integrating Retrieval-Augmented Generation (RAG) with GPT-4o. Medical consultation data were collected from open-source Korean medical resources and public health databases, then preprocessed through cleaning, terminology standardization, de-identification, and classification. The chatbot was fine-tuned for Korean medical dialogues and enhanced with context retention, structured conversation flows, and image-based information extraction. For retrieval, a Korean embedding model (upskyy/bge-m3-korean) and Elastic Search were employed to semantically retrieve relevant medical documents. Metadata Filtering and optimized document chunking improved query precision and retrieval efficiency. Model performance was evaluated using expert-created reference answers, with assessments based on medical accuracy, relevance, and clarity, alongside qualitative clinical validation by medical professionals. RESULTS The proposed model achieved the highest scores in accuracy, relevance, consistency, response time, and cultural appropriateness compared to other models. It showed particularly strong performance in accuracy (4.4) and relevance (4.8), significantly outperforming competing models. In clarity, it also led with a score of 4.9. In qualitative evaluation by medical experts, the model received an 89% approval rate, indicating high trust and reliability in medical consultations. A real-time web service was successfully implemented using an NVIDIA A100 GPU and tools such as Streamlit, Langchain, and Langsmith, enabling fast and accurate health consultations for both the general public and healthcare professionals. CONCLUSIONS This study demonstrates that a RAG-based GPT-4 health chatbot with Metadata Filtering improves the accuracy, relevance, and cultural alignment of Korean medical consultations. The model showed superior performance and clinical validity, though limitations remain in complex reasoning and coverage of lifestyle-related topics. Future efforts should enhance metadata extraction, optimize Korean prompts, and broaden the knowledge base to support more comprehensive healthcare applications.