Benchmarking atom-level explainability against pharmacophore-computed labels in molecular machine learning

Explainability of machine learning models applied to molecular data remains a major challenge, particularly at the atom level, where standardized benchmarks for assessing explanation quality are still largely missing. We introduce a controlled synthetic benchmark constructed from pharmacophore-derived, rule-based reference annotations. The benchmark is based on agreement with predefined geometric constraints that describe spatial arrangements of functional elements within molecules. These rules are derived from multiple independent geometric definitions and serve as a reference for evaluating whether model explanations recover structurally meaningful subsets of atoms. The proposed PharmacoScore quantifies the agreement between atom-level attributions and rule-derived pharmacophore references. We benchmark an initial set of atom-level attribution methods applicable across the selected model classes representing different architectural paradigms, including geometry-agnostic fragment-based approaches and distance-aware transformer models that explicitly encode interatomic geometry. Model explanations are quantitatively evaluated using a unified scoring framework that measures their consistency with predefined geometric constraints. In post-hoc analysis, distance-aware transformer models consistently achieve higher scores, indicating that the highlighted atomic contributions better correspond to expected spatial relationships. This study substantially extends our earlier conference work by introducing a significantly larger and more rigorous dataset, enabling a more reliable evaluation (Sułek et al., 2025). Here, we provide a unified benchmark and evaluation framework for controlled atom-level explainability in molecular machine learning, facilitating systematic comparison between geometry-agnostic and geometry-aware approaches.