What question did this study set out to answer?

To develop an efficient framework for fine-tuning multimodal large language models for medical tasks like question answering and report generation.

April 24, 2026Open Access

PeFoMed: Parameter efficient fine-tuning of multimodal large language models for medical CXR

Key Points

To develop an efficient framework for fine-tuning multimodal large language models for medical tasks like question answering and report generation.
Developed a parameter efficient framework for fine-tuning multimodal models on medical tasks.
Applied the framework to medical visual question answering and medical report generation using public datasets.
Introduced a new evaluation metric combining human ratings and models like GPT-4.
GPT-4-based evaluation showed potential as a scalable method for assessing report quality.
Discrepancies between GPT-4 ratings and traditional lexical similarity metrics were observed.
The study highlights the need for better evaluation methods in generative tasks.

Abstract

Multimodal large language models (MLLMs) represent an evolutionary expansion in the capabilities of traditional large language models, enabling them to tackle challenges that surpass the scope of purely text-based applications. Recent works investigate the adaptation of MLLMs as a universal solution to address medical multi-modal problems as a generative task. In this paper, we propose a parameter efficient framework for fine-tuning MLLMs, specifically validated on medical visual question answering (Med-VQA) and medical report generation (MRG) tasks, using public benchmark datasets. We also introduce an evaluation metric using the 5-point Likert scale and its weighted average value to measure the quality of the generated reports for MRG tasks, where the scale ratings are labelled by both humans manually and the GPT-4 model. We further assess the consistency of performance metrics across traditional measures, GPT-4, and human ratings for both VQA and MRG tasks. The results indicate that GPT-4-based semantic evaluation can provide a scalable supplementary signal for assessing generated outputs, yet they reveal a discrepancy when compared to conventional lexical similarity measurements. This questions the reliability of lexical similarity metrics for evaluating the performance of generative models in Med-VQA and report generation tasks.

Mark Helpful

Bookmark

Relay

View Full Paper