A Feature Matters When Breaking It Hurts: What Permutation Importance Actually Measures

Permutation feature importance is often interpreted as though it reveals which variables matter intrinsically. This paper evaluates a narrower claim: permutation importance measures how much a chosen predictive score changes when an observed feature association is disrupted under a declared permutation rule and evaluation population. Seven tree systems developed across Papers 09–15 are reconstructed without changing their official hyperparameters, representations, stages, or selected endpoints. The study retains the established temporal protocol: 307 vehicles from 1970–1979 form the development period, while 85 vehicles from 1980–1982 form the temporal benchmark. For every frozen model, seven semantic features are permuted independently 200 times within each period. Increase in mean absolute error is the primary estimand, while RMSE and R² provide metric-sensitivity audits. The experiment contains 19,600 feature-wise permutation evaluations and 8,400 grouped evaluations. Weight ranks first in six of the seven frozen systems. AdaBoost.R2 instead ranks horsepower first. Mean pairwise Spearman correlation between temporal importance rankings is 0.702, with a minimum of 0.180 between the Decision Tree and Gradient Boosting. Mean development-to-temporal rank correlation is 0.626. Four of the 49 temporal model-feature mean importances are negative beyond numerical tolerance. Grouped interventions examine size-and-power, time-and-origin, and driving-response information. These groups are conceptual and deliberately overlapping: horsepower belongs to both the size-and-power and driving-response groups. The results show that permutation importance depends on the fitted predictive object, evaluation distribution, scoring rule, perturbation operator, feature-dependence structure, and period being evaluated. It measures predictive reliance under an engineered disruption. It does not establish causal necessity, a unique feature hierarchy, or transportable relevance. The paper introduces the Permutation Reliance Validity Map and includes cross-model comparisons, metric-sensitivity analysis, development-to-temporal transport, grouped perturbations, permutation uncertainty, causal claim boundaries, figures, machine-readable outputs, code, data, and a complete reproducibility package.