Machine learning-driven prediction and optimization of renewable energy integration in multi-regional economic systems: Evidence from China’s carbon markets

China’s commitment to carbon neutrality by 2060 requires precise, regionally differentiated forecasts of renewable energy (RE) penetration and carbon allowance prices to guide investment and policy. This study presents a methodologically rigorous machine learning framework integrating five operational datasets across eight Chinese regional power markets (2019–2023, n = 11,448 training and 2,920 test observations). A leakage-free temporal train/test split, ten-model benchmark with Diebold-Mariano (DM) significance testing, SHAP TreeExplainer interpretability, and quantile-regression prediction intervals collectively address methodological gaps prevalent in prior RE forecasting literature. Building on this protocol we propose Hybrid XGBoost, in which a constriction-factor particle swarm optimizes the boosting hyperparameters, the feature subset and the prediction saturation thresholds jointly under an explicit complexity penalty, and we benchmark it against four hybrid architectures from the literature (GA-XGBoost, PSO-XGBoost, Bayesian-XGBoost and XGBoost-ANN) under identical data, splits and search budgets. Under strict temporal validation, XGBoost achieves R2 = 0.9969 (RMSE = 0.01661) for daily RE penetration rate–significantly outperforming all baselines (DM p < 0.001)–and R2 = 0.9971 (RMSE = 0.420 CNY/tCO2) for carbon allowance prices. Hybrid XGBoost lowers this to R2 = 0.9972 (RMSE = 0.01598), the best of the fifteen models compared and a significant gain over the hyperparameter-only hybrids, whereas tuning hyperparameters alone improved the validation error without improving the held-out year at all. Calibrated 90% prediction intervals attain 0.930 empirical coverage. SHAP analysis identifies 7-day autoregressive persistence and regional CO2 flux as dominant predictors; an ablation study confirms physical features alone yield R2 = 0.9025. A linear programming optimization module translates ML outputs into a regional RE investment allocation–prioritizing Southwest and Northeast China–projected to reduce aggregate carbon intensity by 5.9% against the IEA Statistics baseline, while a non-linear program that lets this response saturate leaves that plan within 0.03 percentage points of optimal but revises the projected reduction to 4.8%, recovering the linear case in the first-order limit, providing a data-driven pathway toward China’s 2030 carbon peak target.