Training visual language models for MRI-based diagnostics

In modern radiology, diagnostic Artificial Intelligence assistants for classifying Magnetic Resonance Imaging (MRI) brain images are typically based on Convolutional Neural Networks (CNNs), which work exclusively with the visual signal and are unable to use structured diagnostic descriptions of pathologies. Vision Language Models (VLMs) open up new possibilities: joint processing of visual and textual data allows for the consideration of the semantic context of pathologies, which is inaccessible to CNN classifiers. The key obstacle to their application remains the high resource intensity of retraining, which is incomparable with the capabilities of a typical clinic. This paper proposes a method for constructing a resource-efficient diagnostic AI assistant based on VLM Gemma3 (4b-it) using low-rank adaptation (LoRA). Retraining was performed on a dataset of 91,777 Fluid-Attenuated Inversion Recovery MRI brain images, distributed across eight pathology classes. Only low-rank adapters were trained — 12.5 % of the model parameters, which allowed training on a single NVIDIA RTX3090 Graphics Processing Unit with 24 GB of Video Random Access Memory without masked data annotation. The radiologist’s expert knowledge was integrated via a system prompt, which assigns the model the role of an experienced neuroradiologist with explicit analysis criteria. On the validation set (9178 examples), the model achieved: Accuracy 0.930, F1-score 0.929, Specificity 0.989, and Cohen’s Kappa 0.915. Comparison with the baseline ConvNeXt-Base classifier on an identical dataset showed the superiority of VLM+LoRA across all metrics: Accuracy increased by 7.4 percentage points, and Cohen’s Kappa by 0.090. The obtained results confirm the practical applicability of the proposed approach for clinics with limited computing resources.