Information criterion for approximation of unnormalized densities

This is the authors' abstract. We don't add key points for this paper.

This paper considers the problem of approximating an unknown density when it can be evaluated up to a normalizing constant at a finite number of points. This density approximation problem is ubiquitous in statistics, such as approximating a posterior density for Bayesian inference and estimating an optimal density for importance sampling. We consider a parametric approximation approach and cast it as a model selection problem to find the best model in pre-specified distribution families (e.g., select the best number of Gaussian mixture components and their parameters). This problem cannot be addressed with traditional approaches that maximize the (marginal) likelihood of a model, for example, using the Akaike information criterion (AIC) or Bayesian information criterion (BIC). We instead aim to minimize the cross-entropy that gauges the deviation of a parametric model from the target density. We propose a novel information criterion called the cross-entropy information criterion (CIC) and prove that the CIC is an asymptotically unbiased estimator of the cross-entropy (up to a multiplicative constant) under some regularity conditions. We propose an iterative method to approximate the target density by minimizing the CIC. We demonstrate how the proposed method selects a parametric model that well approximates the target density through multiple numerical studies in the Supporting Information.