IMG-07. Glio-LLaMA-Vision: A Robust Vision-Language Model for Molecular Status Prediction and Radiology Report Generation in Adult-type Diffuse Gliomas

BACKGROUND To establish a robust vision-language model (“Glio-LLaMA-Vision”) for molecular status prediction and radiology report generation (RRG) in adult-type diffuse gliomas. METHODS Multiparametric MRI data (T1, T2, FLAIR, and postcontrast T1-weighted images) and paired radiology reports (in English) from 1,001 patients with adult-type diffuse gliomas (144 oligodendrogliomas, 157 IDH-mutant astrocytomas, and 700 IDH-wildtype glioblastomas) diagnosed according to the 2021 WHO classification were included in the institutional training set. A vision-language model, Glio-LLaMA-Vision, was developed from LLaMA 3.1 pre-trained on 2.79 million biomedical image-text pairs from PubMed Central and further optimized via fine-tuning from the institutional training set. The performance was validated in 100 patients and 80 patients with paired MRI-radiology reports from an institutional validation set and another tertiary institution, and in 170 and 477 patients with MRI from TCGA and UCSF, respectively. RESULTS In terms of IDH mutation status prediction, Glio-LLaMA-Vision showed an overall performance of area under the curve, accuracy, sensitivity, and specificity of 0.89 (95% confidence interval 0.81-0.95), 86.0%, 84.0%, and 88.0%, respectively. In terms of radiology report generation, the BLEU-1, ROUGE-L, and METEOR scores were 0.49, 0.42, and 0.24, respectively, while the majority (91.3%) of generated reports were considered clinically acceptable. CONCLUSION Glio-LLaMA-Vision shows promising performance in molecular status prediction, and RRG in adult-type diffuse gliomas, and shows potential of clinical assistance.