Beyond Random Splits: Outcome-Prior-Aware Evaluation of Spatial–Temporal Generalization in Civil Engineering AI

Description This repository provides the Python analysis code and supporting documentation for the research paper: Beyond Random Splits: Outcome-Prior-Aware Evaluation of Spatial–Temporal Generalization in Civil Engineering AI Authors: Ehsan Ghaffari and Mohammad KoohiTarget Journal: Journal of Computing in Civil Engineering, American Society of Civil Engineers (ASCE) Overview This study presents a leakage-controlled, deployment-oriented evaluation framework for assessing the generalization and probability reliability of machine-learning models under independent and identically distributed (IID), temporal, spatial, and spatiotemporal distribution shifts. The framework is investigated using two civil-engineering applications: Crash-severity classification: 25,499 police-reported crash records from Razavi Khorasan Province, Iran (2017–2019), with experiments involving multiple model families, controlled feature environments, training-size comparisons, outcome-prior standardization, and calibration diagnostics. Bridge deck-condition prediction: 670,502 two-year bridge-condition pairs derived from the U.S. National Bridge Inventory, covering seven states and multiple evaluation periods. The study examines how deployment conditions influence predictive discrimination, probability calibration, model-selection stability, geographic transfer, and the practical usefulness of predictive models. Repository Contents The code is organized into nine numbered research steps: Crash Calibration Audit: Evaluation of probability reliability and calibration under distribution shift. Crash Model Comparison: Comparison of model families and generalization behavior. Outcome-Prior Analysis: Investigation of outcome-prior standardization and distribution-shift diagnostics. Bridge Data Preparation: Processing and preparation of National Bridge Inventory records. Bridge Modeling: Model training and evaluation for bridge deck-condition prediction. Bridge Adaptation: Evaluation of outcome-prior adaptation and labeled-audit strategies. Geographic Exposure Comparison: Matched-training-size analysis of seen-versus-unseen geographic domains. Crash Prior Adaptation: Post-hoc expectation–maximization prior-adjustment experiments. Crash Recalibration: Temperature-scaling experiments using different labeled-data budgets. The repository also includes dependency specifications, methodological documentation, setup instructions, and validation information. Data Availability and Reproducibility This is a code-only research release. The original Iranian police crash records and individual-level predictions are excluded because of data-access and confidentiality restrictions. Researchers seeking access to restricted data must obtain authorization from the original data provider. The bridge replication uses National Bridge Inventory data. The repository documents the required inputs and analysis procedures. The included scripts support methodological transparency and authorized reproduction of the reported analyses. However, the archive does not contain all original campaign source modules or restricted inputs and therefore should not be interpreted as a fully self-contained reproduction of every experiment. Software Language: Python Research areas: Civil engineering AI, machine learning, domain generalization, distribution shift, model calibration, transportation safety, and bridge asset management. Citation If you use this code, please cite this Zenodo repository and the associated research article when published.