An experimental and theoretical comparison of model selection methods

In the model selection problem... The goal of this paper is to provide such a comparison, and more importantly, to describe the general conclusions to which it has led. Relying on evidence that is approximately equally divided between controlled experimental results and related formal analysis, we compare three well-known model selection algorithms and attempt to identify their relative and absolute strengths and weaknesses, and we provide some general methods for analyzing the behavior and performance of model selection algorithms. Our hope is that these results will help the informed practitioner make an educated choice of model selection algorithm (perhaps based in part on some known properties of the model selection problem confronting them). The summary of the paper follows. In Section 2, we provide a formalization of the model selection problem. In this formalization, we isolate the problem of choosing the appropriate complexity...