Bayesian Supervised Learning Methods: Regularization and Predictive Uncertainty

Bayesian formulations of supervised learning methods differ in their prior distributions, inferential procedures, and treatment of predictive uncertainty. In this paper, we present a review of regularized regression, nearest-neighbor methods, support vector machines, tree ensembles, generalized additive models, Gaussian processes, and neural networks. The main objective is to examine the relationship between classical estimation procedures and their Bayesian formulations, with particular attention to posterior prediction. We consider likelihood- and loss-based updating, fixed and estimated hyperparameters, posterior integration, and computational approximations. Maximum a posteriori (MAP) estimation is distinguished from posterior uncertainty, empirical Bayes from cross-validation-based tuning, and generalized updating from approximations to a likelihood-based posterior. The methods are compared in terms of calibration, prior sensitivity, model misspecification, interpretability, and scalability. Available software is described in a dated inventory. An illustrative comparison is performed in R using a real data set on concrete compressive strength, with common composition-grouped validation folds. Predictive accuracy, proper scoring rules, calibration, interval coverage, and computational cost are evaluated, and the approximation and numerical diagnostics are reported for each procedure. The discussion and numerical results indicate that predictive uncertainty depends on both model specification and inference; a Bayesian formulation alone does not ensure empirical calibration.