Leveraging Open-Source Large Language Models for Non-Technical Researchers for Data Visualization: Performance Analysis in Secondary Data Use Systems (Preprint)

BACKGROUND Medical institutions increasingly prioritize the secondary use of clinical data for research purposes, balancing research advancements with strict data protection regulations. In this context, the German University Medical Center Hamburg-Eppendorf (UKE) has developed the data hotel. This platform enables researchers to access pseudonymized clinical data for retrospective studies without requiring additional ethical or legal clearances. OBJECTIVE To enhance the platform’s usability, this study evaluated the integration of a large language model (LLM)-based code chatbot designed to generate Python code for data visualization and analysis, specifically addressing challenges posed by the heterogeneity and complexity of clinical datasets. METHODS Three locally installed, open-source LLMs – Mistral-7B-Instruct, Llama3-8B-Instruct, and CodeLlama-7B-Instruct – were assessed for their ability to answer basic and complex research queries using different prompting techniques, including zero-shot, one-shot, few-shot, and chain-of-thought prompting. Performance, including reproducibility and repeatability, was measured against ground truth by using different phrasing styles. Phrasings were also translated into German to evaluate the model’s language dependency. RESULTS Llama3, especially when paired with few-shot prompting, outperformed the other models in generating accurate and complete code for basic queries across two datasets. However, performance declined for complex queries, with accuracy being significantly influenced by the dataset's structure and quality. German prompts yielded lower scores compared to English, highlighting the limitations of the linguistic model. The study identified the need for better data preparation, tailored prompting strategies, and advanced software architectural approaches to address dataset-specific challenges. CONCLUSIONS This work demonstrates the feasibility of integrating LLM-based tools into clinical data platforms, empowering non-technical researchers with a scalable approach for secondary data analysis. It emphasizes the importance of data quality and contextual understanding. Future directions include expanding prompt libraries, incorporating advanced visualization tools, implementing agentic systems, and exploring higher-parameter models to improve versatility and robustness.