Generative explainers in Spanish healthcare prognosis: a novel assessment framework

Abstract Preventive medicine aims to detect potential health risks early, thereby reducing morbidity and healthcare expenses. Beyond the mere anticipation of possible diagnoses, there is a demand for systems that justify predictions with clinical evidence and allow a quantitative assessment of the quality of the explanation. In this work, we present a novel approach: first, to address the prognosis of subsequent diagnoses in Spanish clinical data and, second, to generate explanations for each estimated diagnosis using generative models. By structuring patient data with the ICD-10 coding standard, and with the aid of cross-lingual data augmentation, substantial prognosis estimation improvements were achieved. We also put special attention to the assessment of the quality of the explanations generated. In this line, we propose the FIR framework (i.e. faithfulness, interpretability, robustness), in an attempt to seize the ability of each explanation to address core clinical questions relevant to the estimated ICD-10 diagnoses. Combining ICD-10 timelines coming from Osa (Spanish EHRs dataset) with MIMIC-IV (English EHRs dataset) significantly improved our Disease Risk Identifier’s accuracy (by 11% in Spanish and 6% in English). Two LLMs (Mixtral and BioMistral) were used to generate explanations. Mixtral achieved higher faithfulness and interpretability (FidIn score of 74.59) compared to BioMistral (45.25), yet both maintained stable performance under minor textual perturbations. These findings demonstrate that leveraging ICD-10 diagnoses timelines coming from different data sources notably enhances next diagnosis predictions for Spanish EHRs. The proposed Disease Risk Explainer successfully produces clinically relevant textual justifications, and the FIR framework provides an objective metric to measure explanation quality.