Sensitivity analysis of Welch’s t-test

Welch t-test is the parametric test for comparing means between two independent groups without assuming equal population variances. This statistic is robust for testing the mean equality when homogeneity assumption is not satisfied, but Welch test is not always robust. When multiple problems such as the distribution is non-normal, variance is heterogeneous and unequal size of groups occur simultaneously, the Type I error will inflate. In this study, various conditions such as sample sizes, type of distributions and unequal group variances were manipulated to investigate on the non robust conditions of Welch test. The Type I error rates and power of the test for different design specifications were obtained and compared. The results indicated that this test did not perform well under non-normal distributions especially when group sizes and unequal group variances are inversely associated or negatively paired. The estimated Type I error inflated as the power of the test improved.