Variable importance is variance, not effect size: a caution and a prevalence screen

Mean-decrease-in-impurity (MDI) variable importance is the default, cheapest way to read a random forest, and it is routinely misread as a per-unit effect size, as if it were comparable to a linear-model coefficient. In additive settings MDI instead estimates each feature’s variance contribution. We make the practical consequences concrete through five controlled experiments: impurity importance (i) grows with the square of a linear coefficient, not the coefficient; (ii) assigns an interaction’s variance to the interacting variables; and (iii) gives provably identical scores to a genuinely non-linear effect and to a linear effect on a skewed feature, because the two contribute identical variance. These are not artefacts of the well-known bias toward high-cardinality, continuous, or correlated features: they arise among features that are continuous, uncorrelated, and identically distributed, and they are not specific to impurity: permutation importance behaves the same way, and standardised regression coefficients spread similarly. No marginal importance measure separates an effect’s shape from a feature’s distribution; partial-dependence plots do. Finally, an LLM-assisted screen of the ranger-citing literature, an unvalidated estimate rather than an audit, suggests that about one in six papers that interpret an impurity ranking rely on it without such corroboration, including heavily-cited work