Multiple Comparisons of Log-Likelihoods and Combining Nonnested Models With Applications to Phylogenetic Tree Selection

Abstract We consider multiple comparisons of log-likelihood's to take account of the multiplicity of testings in selection of nonnested models. A resampling version of the Gupta procedure for the selection problem is used to obtain a set of good models, which are not significantly worse than the maximum likelihood model; i.e., a confidence set of models. Our method is to test which model is better than the other, while the object of the classical testing methods is to find the correct model. Thus the null hypotheses behind these two approaches are very different. Our method and the other commonly used approaches, such as the approximate Bayesian posterior, the bootstrap selection probability, and the LR test against the full model, are applied to the selection of molecular phylogenetic tree of mammal species. Tree selection is a version of the model-based clustering, which is an example of nonnested model selection. It is shown that the structure of the tree selection problem is equivalent to that of the variable selection problem of the multiple regression with some constraints on the combinations of the variables. It turns out that the LR test rejects all the possible trees because of the misspecification of the models, whereas our method gives a reasonable confidence set. For a better understanding of the uncertainty in the selection, we combine the maximum likelihood estimates (MLE's) of the trees to obtain the full model that includes the trees as the submodels by using a linear approximation of the parametric models. The MLE of the phylogeny is then represented as a network of species rather than a tree. A geometrical interpretation of the problem is also discussed. Keywords: Model selectionMisspecificationRanking and selectionMultiple comparisonsSelection biasVariable selectionArtificial regressionMolecular evolution ACKNOWLEDGMENT The connection between the trees and the variable selection was developed when I was visiting J. Felsenstein as a post-doctoral fellow. The biological arguments are mostly given by M. Hasegawa. This version of the manuscript is written during my visit to B. Efron. I thank them as well as Y. Cao, H. Kishino, Z. Yang, P. Waddell, N. Goldman, and S. Holmes for the helpful discussions in phylogenetic problems. I thank the anonymous reviewer for the constructive comments.