MatUQ: a benchmark for uncertainty-aware out-of-distribution materials property prediction with graph neural networks

Reliable uncertainty quantification (UQ) for graph neural networks (GNNs) under out-of-distribution (OOD) shifts remains insufficiently characterized in materials discovery. Existing benchmarks based on random splits can overestimate model reliability by underrepresenting structural extrapolation challenges. Here we introduce MatUQ, a benchmark built on structure-aware Smooth Overlap of Atomic Positions Leave-One-Cluster-Out (SOAP-LOCO) splitting, together with a training protocol that combines Deep Evidential Regression (DER) with dropout regularization, for evaluating GNN reliability under structural distribution shifts. Through systematic experiments spanning six materials datasets, twelve GNN architectures, and eight UQ strategies, we find that predictive accuracy and uncertainty quality are distinct capabilities that tend to decouple under OOD evaluation, and that uncertainty-metric leadership is largely non-transferable across datasets and target properties. We further find that as training data become scarce, the optimal strategy shifts from evidential-containing hybrids toward pure ensemble variance across all evaluated architectures, with the architecture holding the distributional optimum shifting correspondingly. Standalone evidential regression rarely attains per-model optima and benefits from hybrid pairing only under specific combinations of data density, inductive bias, and target metric. Monte Carlo dropout is less effective as a standalone uncertainty estimator under the tested configurations but can contribute within hybrid schemes. Overall, MatUQ provides a more rigorous benchmark for assessing uncertainty-aware GNNs in OOD materials discovery.