Trustworthy machine learning of diffractograms to prevent silent failure of apparently accurate models

For automated analysis of diffractograms, it seems straight forward to use machine-learning models developed for images, which can indeed generate highly accurate predictions. However, the fundamental difference between images and diffractograms can introduce hidden vulnerabilities. As a case study, we analyze - and -phase fractions in Ti–6Al–4V using two independent synchrotron 2D X-ray diffractogram datasets. We analyze the performance of two neural networks: “IE,” pre-trained only on ImageNet (a database of natural images), and “ISE,” additionally trained on a large corpus of simulated X-ray diffractograms. Subsequently fine-tuned with just 250 experimental diffractograms, both models achieve similarly high predictive accuracy. However, robustness tests and attention mapping reveal fundamentally different internal logic. IE suffers from “surface-texture bias,” predicting phase fractions from irrelevant background and pixel noise. ISE, in contrast, evaluates physically relevant features – Bragg reflections – in the diffractograms. Accordingly, prediction accuracy alone does not validate machine-learning models. Trustworthy models for diffractogram analysis require physics-driven pre-training. Machine-learning models can accurately predict outcomes from diffractograms, but differences between images and diffractograms may introduce vulnerabilities. Here, the authors compare neural networks trained on natural images and simulated diffractograms, revealing that physics-driven pre-training enhances model robustness by focusing on relevant features, underscoring the need for domain-specific training in scientific applications.