Comparative Analysis of Conformal Prediction: Split, Full, and Adaptive Approaches for Statistical and Neural Network Models

ABSTRACT: Conformal prediction (CP) is a framework that provides uncertainty quantification output as valid marginal coverage for predictive models. At present, the main methods used are divided into Bayesian methods and statistical inference method. Among the statistical inference methods, split, full and adaptive conformal prediction are the basic methods. Although there are numerous variations of these methods, a clear comparison is lacking. In this paper, three basic conformal prediction methods are compared on low-dimensional and high-dimensional dataset to illustrate the advantages and disadvantages of each method. The experiment shows that split conformal prediction performs stable coverage but holds data partition as key issue to solve; Expected coverage could not be achieved by Full conformal though it can decrease the prediction interval; Adaptive conformal prediction faces the quantile distribution deviation of complex model. This paper also illustrate the direction of future research.