New Two-Stage and Sequential Procedures for Selecting the Best Simulated System

Standard “indifference-zone” procedures that allocate computer resources to infer the best of a finite set of simulated systems are designed with a statistically conservative, least favorable configuration assumption consider the probability of correct selection (but not the opportunity cost) and assume that the cost of simulating each system is the same. Recent Bayesian work considers opportunity cost and shows that an average case analysis may be less conservative but assumes a known output variance, an assumption that typically is violated in simulation. This paper presents new two-stage and sequential selection procedures that integrate attractive features of both lines of research. They are derived assuming that the simulation output is normally distributed with unknown mean and variance that may differ for each system. We permit the reduction of either opportunity cost loss or the probability of incorrect selection and allow for different replication costs for each system. The generality of our formulation comes at the expense of difficulty in obtaining exact closed-form solutions. We therefore derive a bound for the expected loss associated potentially incorrect selections, then asymptotically minimize that bound. Theoretical and empirical results indicate that our approach compares favorably with indifference-zone procedures.