Bayesian neural networks (BNNs) offer an elegant and promising approach to deciding whether the predictions of a neural network are trustworthy by allowing the estimation of predictive distributions. However, training and prediction can only be performed approximately, and state-of-the-art approximation methods are known to frequently provide inaccurate uncertainty estimations, thus limiting the broad application of neural networks. To remedy this, we define criteria for trustworthy predictions and propose a new approach capable of identifying input space regions with trustworthy predictions. For this, we use statistical hypothesis testing on the BNN’s predictions and point out some connections to previously known calibration and uncertainty estimation metrics. We demonstrate our method using several state-of-the-art approximate inference methods on two single-input, single-output regression tasks. Our results show that the proposed approach identifies input space regions with well-calibrated uncertainty predictions while providing valuable insights into the test statistics of the underlying distributions.
Walker et al. (Sun,) studied this question.