T-Friedman Test: A New Statistical Test for Multiple Comparison with an Adjustable Conservativeness Measure

Abstract To prove that a certain algorithm is superior to the benchmark algorithms, the statistical hypothesis tests are commonly adopted with experimental results on a number of datasets. Some statistical hypothesis tests draw statistical test results more conservative than the others, while it is not yet possible to characterize quantitatively the degree of conservativeness of such a statistical test. On the basis of the existing nonparametric statistical tests, this paper proposes a new statistical test for multiple comparison which is named as t-Friedman test. T-Friedman test combines t test with Friedman test for multiple comparison. The confidence level of the t test is adopted as a measure of conservativeness of the proposed t-Friedman test. A bigger confidence level infers a higher degree of conservativeness, and vice versa. Based on the synthetic results generated by Monte Carlo simulations with predefined distributions, the performance of several state-of-the-art multiple comparison tests and post hoc procedures are first qualitatively analyzed. The influences of the type of predefined distribution, the number of benchmark algorithms and the number of datasets are explored in the experiments. The conservativeness measure of the proposed method is also validated and verified in the experiments. Finally, some suggestions for the application of these nonparametric statistical tests are provided.