Validation-Generalization Asymmetry in Metaheuristic CNN Forecasting: An Extremely Small Time-Series Case

Hyperparameter optimization can reduce validation error while worsening future performance when data are extremely scarce. This study examines that risk in a public railway-mileage implementation workbook containing 17 ordered five-lag input-target rows, of which 14 form the development set and three form a locked outer test. No independently verified 22-point raw series was available; exact archived rows are therefore the primary data, while an overlap-consistent inferred chronology is used only for sensitivity analysis. WOA-CNN and BWO-CNN were evaluated through 10 complete HPO searches under shuffled and rolling-origin validation. Five-search purged runs were used as a stress-test sensitivity, while a fixed Manual CNN served as the matched reference. Median locked-test MAPE was 3.75% for WOA-Random, 2.56% for BWO-Random, 4.16% for WOA-Rolling, and 4.51% for BWO-Rolling, compared with a matched Manual CNN reference mean of 2.54%. The four HPO procedures beat that reference in only 3/10, 5/10, 2/10, and 1/10 searches, respectively. A mean-increment baseline obtained 0.63% MAPE. A separate benchmark of 160 independent simulated series examines sample-size and validation-selection mechanisms under four pre-specified data-generating processes. Taken together, the two analyses characterize how validation design and data availability affect model-selection stability in extremely small time-series settings.