While many advanced statistical methods for the design of experiments exist, it is still typical for physical experiments to be performed adaptively based on human intuition. As a consequence, experimental resources are wasted on sub-optimal experimental designs. Conversely, in the simulation-based design community, Bayesian optimization (BO) is often used to adaptively and efficiently identify the global optimum of a response surface. However, adopting these methods directly for the optimization of physical experiments is problematic due to the existence of experimental noise and the typically more stringent constraints on the experimental budget. Consequently, many simplifying assumptions need to be made in the BO framework, and it is currently not fully understood how these assumptions influence the performance of the method and the optimality of the final design. In this paper, we present an experimental study to investigate the influence of the controllable (e. g. , number of samples, acquisition function, and covariance function) and noise factors (e. g. , problem dimensionality, experimental noise magnitude, and experimental noise form) on the efficiency of the BO framework. The findings in this study include, that the Mat\'er covariance function shows superior performance over all test problems and that the available experimental budget is most consequential when selecting the other settings of the BO scheme. With this study, we enable designers to make more efficient use of their physical experiments and provide insight into the use of BO with intrinsically noisy training data.
Anton van Beek (Fri,) studied this question.