Understanding the Variability of Simulation Results: An Empirical Study

This paper describes an empirical study of the variability of simulation estimates, which produced some interesting new results. Simulation estimates are determined by the input samples. Any input sample can be divided into two basic features: the set of input values and their sequence. Based on this idea, the individual contribution of each feature to the estimates' variance can be empirically studied. This is done following a two-way factorial experiment. Using the standard random sampling approach, the set of values was found to affect most simulation estimates in a common way, and to play a relevant role in their variability. The sequence effect, however, was found to be problem-dependent. Apart from providing a better understanding of the estimates' variability, this study contributes to the proposal of a new sampling approach in simulation: descriptive sampling.