Determining the empirical probability function of random samples drawn from a continuous distribution using information theory

Abstract The empirical probability of a set of random samples drawn from a continuous distribution is often estimated using a histogram. Depending on the choice of the number of bins the probability distribution function can vary. In this paper the optimum number of bins used to estimate its histogram is chosen using information theory based on Shannon entropy. It involves considering the trade-off between capturing enough detail or information and avoiding excessive uncertainty or disorder. It maximizes the gain in information content when the number of bins is varied. The method is validated by evaluating the Kullback-Leibler Divergence (KL Divergence) for random samples drawn from a number of continuous distributions. The results are also compared with other methods of evaluating the optimum number of bins. The method performs very well and provides a novel way to determine the underlying probability structure of the continuous distribution from which the random samples are drawn.