Multimodal brain lesion classification approach through MRI radiomics and OCR-recovered MRS metabolites with explainable AI

This work presents a multimodal machine learning pipeline for brain lesion classification, fusing structural radiomic features from multi-sequence MRI (T2-FLAIR, T1-contrast) with metabolic features recovered from Magnetic Resonance Spectroscopy (MRS) screenshots via a custom Optical Character Recognition (OCR) pipeline. The study draws on a clinical cohort of 55 patients (17 non-neoplastic, 38 neoplastic). MRI radiomics span first-order intensity statistics, Grey-Level Co-occurrence Matrix (GLCM) texture, morphological shape, and contrast-enhancement ratios, while MRS metabolites (Cho, Cr, NAA, and their ratios) are lifted from rendered spectra through Tesseract OCR with multi-region cropping and Lanczos upsampling. Features are evaluated under a leakage-controlled Leave-One-Out Cross-Validation (LOOCV) protocol with in-fold imputation, feature selection and normalization, using the classifier’s default decision threshold. The primary, bias-corrected performance estimate comes from nested LOOCV, in which classifier family and feature count are chosen using training data only: the multimodal pipeline reaches 82.0% accuracy. The best fixed configuration (Random Forest, k = 3) reaches 85.5% accuracy (47/55) and AUC = 0.82, but because it is selected on the same data it is scored on, it is reported as an exploratory, optimistic figure. The comparison model, which uses MRI-derived features without OCR-recovered metabolites, reaches 78.2% at its own best configuration, so the best-configuration gain is +7.3 percentage points and the nested estimate exceeds that baseline by +3.8 points. The two strongest discriminators are log NAA ( F = 16.01) and log Cho ( F = 12.89). The best-configuration result holds under 1,000-permutation testing (random-label mean 55.0%) and McNemar’s exact test ( p = 0.05, borderline). At N = 55, and with diagnostic labels that themselves drew on MRS, these results stand as proof-of-concept: OCR-recovered MRS metabolites are informative for reproducing expert diagnostic judgment beyond what MRI-derived features supply alone.