A framework for evaluation and requirement extraction for fine-tuning of Large Language Models in multimodal medical diagnosis

Large language models constitute a breakthrough state-of-the-art Artificial Intelligence technology which is rapidly evolving and promises to aid in medical diagnosis. In this study, we propose a novel evaluation framework for extraction of fine-tuning requirements based on the Objective Structured Clinical Examinations (OSCE) that can increase LLM potential and applicability. We developed an OSCE based evaluation meta-framework leveraging IoT-based data retrieval with a two-step approach designed to analyze and guide improvement of LLMs in multimodal medical diagnosis: (1) structured interaction evaluation and (2) domain-specific analysis of extracted data. Using Image-Metadata Analysis (IMA), Named Entity Recognition (NER), and Knowledge Graphs (KG), this framework identifies image domains, extracts relevant entities, and assesses connections in KGs. These methods collectively reveal areas for improvement, guiding fine-tuning to enhance diagnostic accuracy and contextual understanding in medical applications. Using this paradigm, (1) we evaluate the correctness and accuracy of generated medical diagnosis with publicly available multimodal-multiple-choice-questions in the vast domain of General Pathology and (2) proceed to the domain-specific analysis. We identify and visualize the model performance across specific organs, diseases, and pathological themes, detecting areas of lower accuracy, such as in cardiovascular conditions like atherosclerosis. This targeted approach enables precision-focused fine-tuning, applying additional data to specific weaknesses rather than a broad, generalized tuning across all pathology. Our framework’s primary contribution is its OSCE-inspired ability to dynamically identify and target under-performing areas within a broad domain, enhancing fine-tuning efficiency and diagnostic accuracy in a resource-effective and iterative manner removing the dependency on bulk adjustments, making it particularly suitable for sensitive applications where precision and resource efficiency are essential, such as in medical diagnostics. • Systematic Fine-Tuning Guidance: A structured two-step framework for identifying and addressing LLM weaknesses in multimodal medical diagnosis. • Multimodal Evaluation: Integrates textual and visual data to enhance real-world diagnostic accuracy, particularly in pathology. • Performance & Impact: Applied to GPT-4V, achieving 84% accuracy across 79 image-based medical multiple-choice questions. Targeted fine-tuning based on identified weaknesses is expected to yield substantial accuracy improvements. • Precision-Driven Enhancement: Uses Image Metadata Analysis (IMA), Named Entity Recognition (NER), and Knowledge Graphs (KG) to pinpoint areas for improvement, optimizing fine-tuning efficiency. • Broad Applicability: Adaptable across different LLMs and medical subdomains, with potential for generalization to other multimodal AI applications.