Comparing the performance of randomization tests and traditional tests:A simulation study

Abstract

Being non parametric in nature, the randomization tests (RTs) differ from the parametric statistical tests in many aspects and are often assumed to be more robust than parametric tests when their assumptions are violated. However, this ideology lacks sufficient evidence and the virtues of the RTs continue to be debated in the literature often with different conclusions.As a result researchers are often reluctant to employ RTs which are different from status quo and opt to use the traditionaltests, regardless of the characteristics of their data. Hence this study compares the robustness, in terms of type I error rate and power, of the most widely used classical parametric tests; pooled t test, unpooled t test, paired t test and one way ANOVA F test with their respective randomization counterpart using simulations under several trial conditions. The results concluded that, although the RTs are more robust in the presence of certain parametric assumption violations, this should not be a general rule and hence should only be used under the appropriate conditions for each test as demonstrated.