The Split Is Part of the Claim: What Data Partitioning Actually Measures

An out-of-sample result does not demonstrate generalization by itself. It demonstrates performance under the partition mechanism that produced it. When a validation design mixes observations from different periods, it may estimate interpolation within a pooled historical population rather than transport to a genuinely later environment. This paper introduces the Generalization Validity Map, a seven-level framework connecting independence, representativeness, temporality, selection integrity, estimation, transport, and decision validity. Its central rule is that passing an evaluation does not establish generalization beyond the world represented by that evaluation. The framework is evaluated using 392 vehicles from the UCI Auto MPG dataset. Model years 1970–1979, comprising 307 vehicles, form the development period, while model years 1980–1982, comprising 85 vehicles, serve as a temporally separated final benchmark. The estimator, feature representation, preprocessing pipeline, and performance metric remain fixed so that differences in estimated performance can be attributed primarily to the partition protocol. Six evaluation protocols are compared: random holdout, year-stratified random holdout, random five-fold cross-validation, grouped five-fold cross-validation, leave-one-year-out validation, and expanding temporal validation. The experiment includes 200 random holdouts, 200 year-stratified holdouts, 40 repeated five-fold evaluations, and six historical pseudo-deployments in which models are trained using only the available past and evaluated on the subsequent year. Across the pseudo-deployments, random holdout underestimates future MAE by an average of 0.954 mpg and is optimistic in five of six cases. Random five-fold cross-validation produces a nearly identical average optimism gap of 0.956 mpg and is also optimistic in five of six cases. Leave-one-year-out validation has an average optimism gap of 0.609 mpg. Expanding temporal validation is slightly conservative on average, overestimating future error by 0.165 mpg. Grouped five-fold validation is applicable to five of the six pseudo-deployments and is optimistic in three of those five cases. It cannot be applied to the earliest pseudo-deployment because the available history contains only four distinct year groups. On the frozen 1980–1982 temporal benchmark, the fixed Ridge model achieves an MAE of 3.79 mpg, RMSE of 4.68 mpg, and R² of 0.392. Although this substantially outperforms the development-mean baseline MAE of 11.04 mpg, the observed test error remains 1.38 mpg higher than the expanding-temporal development estimate and more than 2 mpg higher than the principal random-protocol estimates. The results do not establish that temporal validation is universally superior. Instead, they demonstrate that each partition protocol supports a different generalization claim. Random partitioning can be appropriate when deployment reproduces the same exchangeable population, but it becomes misleading when it mixes historical periods and permits an evaluation geometry that cannot exist during forward deployment. The record includes a complete reproducibility package containing source code, data, protocol specifications, fold-level metrics, pseudo-deployment results, frozen-test predictions, figures, structured outputs, integrity manifests, and checksums.