Every Split Changes the Model: What a Decision Tree Actually Learns

A decision tree is often interpreted as a transparent representation of the structure contained in a dataset. However, every split is a locally selected decision conditioned by the available sample, the splitting criterion, and the structural restrictions imposed on the model. A readable tree is therefore not necessarily stable, generalizable, or causally meaningful. This paper introduces the Tree Validity Map, a framework for separating predictive fit, temporal transport, structural stability, interpretability, and causal meaning in decision-tree analysis. The framework examines how local reductions in error become a global prediction function and how depth, leaf size, pruning, sampling variation, and predictor importance affect the conclusions drawn from a fitted tree. The empirical study uses 392 vehicles from the UCI Auto MPG dataset. Model years 1970–1979, comprising 307 vehicles, form the development period, while model years 1980–1982, comprising 85 vehicles, serve as a temporally separated benchmark. Decision-tree configurations of varying depth and structural regularization are evaluated under a common preprocessing and temporal-validation protocol. A Ridge regression model is included as a structured linear comparator. The unrestricted decision tree reaches a training MAE of 0.000 mpg, using 239 leaves and a maximum depth of 15. Its temporal-benchmark MAE is 3.419 mpg, with a maximum absolute error of 18.2 mpg. A cost-complexity-pruned tree selected exclusively within the development period contains 53 leaves and records a temporal MAE of 3.914 mpg and a maximum absolute error of 17.1 mpg. In this experiment, pruning substantially reduces structural complexity but does not improve temporal MAE relative to the unrestricted tree. Structural stability is examined using 500 bootstrap resamples. Across those resamples, 43 distinct root rules appear, and the most frequent root rule occurs in only 27.8% of fitted trees. The median standard deviation of benchmark predictions across bootstrap trees is 2.746 mpg. These results demonstrate that the apparent readability of an individual fitted tree does not establish that its structure is uniquely determined by the data. A complementary leave-one-out perturbation audit removes each development observation individually across 200 controlled deletions. None of these perturbations changes the root rule of the reference tree, yet 41% materially alter some part of the predictive function. This shows that stability at the top of a tree can coexist with consequential sensitivity in lower branches. Predictor importance is evaluated using both mean decrease in impurity and temporal-benchmark permutation importance. The two methods produce different interpretations because impurity importance measures reductions achieved during development, whereas permutation importance measures the predictive contribution expressed on the temporal benchmark. A route-level audit explains why model_year and origin_3 retain positive impurity importance while producing exactly zero permutation importance on the benchmark. For model_year, all benchmark observations remain on the same side of every reached year threshold under permutation. For origin_3, no benchmark observation reaches the node containing its split. Consequently, neither terminal assignments nor predictions change across the permutation runs. Temporal prediction bias is defined as the mean of predicted minus observed values. Negative values therefore indicate average underprediction, while positive values indicate average overprediction. This convention supports a direct comparison of error direction between the fitted trees and Ridge regression. The results show that interpolation, temporal transport, structural stability, predictive importance, and causal interpretation are distinct properties. A tree can fit the development sample perfectly while remaining temporally imperfect, retain a stable root while changing materially in lower branches, and assign predictive importance without establishing a causal relationship. The central conclusion is that readable is not the same as stable, stable is not the same as predictive, and predictive is not the same as causal. The record includes a complete reproducibility package containing source code, data, structured results, depth and pruning experiments, bootstrap stability analyses, leave-one-out perturbation audits, predictor-importance comparisons, route-level permutation audits, figures, manifests, and SHA-256 checksums.