The Benchmark Remembers Every Question: What Seven Tree Systems Actually Taught Us

Paper 16 of the series “Artificial Intelligence: From Foundations to Agents.” This paper closes the tree-model subseries developed across Papers 09–15. It synthesizes seven systems evaluated under the same historical protocol: a single decision tree, Random Forest, Gradient Boosting, an educational exact-greedy XGBoost mechanism, CatBoost, LightGBM, and AdaBoost.R2. It does not introduce an eighth model or perform a new model-selection competition. The study introduces the Cumulative Benchmark Validity Map, which separates prespecified selection, official temporal endpoints, retrospective mechanism audits, complete-refit sensitivity, representation differences, accumulated benchmark exposure, and the claims that remain supportable after repeated inspection. All seven studies use the same 392 complete Auto MPG observations. The development period contains 307 vehicles from 1970–1979, with internal selection performed through the 1970–1976 versus 1977–1979 division. The frozen temporal benchmark contains 85 vehicles from 1980–1982. The official comparison retains one prespecified endpoint from each paper. The single decision tree records the lowest official temporal MAE at 3.419 mpg, followed by XGBoost at 3.471 mpg, LightGBM at 3.611 mpg, CatBoost at 3.737 mpg, Gradient Boosting at 3.945 mpg, AdaBoost.R2 at 3.971 mpg, and Random Forest at 4.133 mpg. This ranking is descriptive under the declared procedures and does not establish a universal hierarchy of model families. Model complexity does not produce monotonically better temporal transport. The standalone tree ranks first, Random Forest ranks last, and the boosting systems occupy the middle. All seven systems underpredict the later period, with temporal bias ranging from −2.474 to −3.521 mpg. The shared direction of error points toward temporal shift and protocol scope rather than toward a failure attributable to one optimizer. Accuracy and complete-refit stability also remain separate properties. Median benchmark prediction standard deviation ranges from 0.636 mpg for Random Forest to 1.460 mpg for AdaBoost.R2. Across the seven papers, 1,600 complete development-bootstrap refits were scored on the temporal observations. The synthesis also constructs an explicit exposure ledger. The 1980–1982 benchmark appears in 2,612 recorded benchmark-visible metric rows: seven official endpoints, the complexity and tree-count paths from Papers 09 and 10, five 500-stage boosting trajectories, and the post-selection audit grids from Papers 11–15. These rows are not independent statistical tests, and the count is not used to calculate an optimism correction. It documents how extensively the benchmark has been inspected. Retrospective findings—including stage 485 for Gradient Boosting, stage 299 for XGBoost, max_bin=15 for LightGBM, and depth 3 for AdaBoost.R2—remain diagnostic. They may generate hypotheses for future work, but they cannot replace the official endpoints after the benchmark has revealed them. The central conclusion is: “The benchmark remembers every question. Once it becomes part of the conversation, the next strong claim needs new evidence.” The deposit contains the complete paper, an editable document, and a reproducibility package containing the machine-readable system comparison, synthesis tables, benchmark-exposure ledger, source-generated figures, verification script, dependency specifications, and SHA-256 integrity manifest.