Multiple comparisons and effect size: Statistical recommendations for authors planning to submit an article to Allergy

This article deals with biostatistical recommendations for authors, namely related to two aspects. One of these aspects concerns multiple comparisons, that is the importance of choosing an appropriate post-hoc test to illustrate the actual research results obtained, which are not, for example, due to chance. The second aspect concerns the magnitude of the effect, the various measures to assess it so as to be able to show the practical significance of the results obtained by the authors, not just the p-value, which in itself very often adds little to the world of science. We use analysis of variance (ANOVA) when we test differences between two or more groups of measurements.1 Unfortunately, one of the most common mistakes made by authors is to perform a series of t-student tests without any correction for multiple comparisons. Often this is due to a lack of related knowledge, or to the idea that there is a greater chance of statistically significant differences. However, it should be borne in mind that the more comparisons are made, the more likely it is to obtain random differences between the averages. The probability of making an error of the first type, that is rejecting the null hypothesis despite the fact that it is true, increases. A false alternative hypothesis is then accepted. This is the so-called cumulative probability of error of the first type. Obtaining a significant result can be accidental. When an ANOVA shows us the presence of statistically significant differences, an appropriate post-hoc test should be applied. Unfortunately, one of the main mistakes made by the authors is the wrong choice of such a test or, for example, the automatic application of the Bonferroni post-hoc test. The reason for this is often a lack of knowledge of the names and purposes of using the other post-hoc tests, or a greater desire for statistically significant differences.2 Most post-hoc tests assume a correction for, among other things, the number of comparisons performed. This ensures that the probability level of error of the first type does not exceed 0.05. The balance between power and type I error rate must be maintained. If the post-hoc test is too conservative, an error of the first kind is unlikely to occur. The individual post-hoc tests were designed for settings where they have maximum power. The selected post-hoc tests and their description are included in Table 1. Based on this description, it is recommended that authors include a brief description in their submitted manuscripts as to why this and not another post-hoc test was used. This increases the reliability of the research results obtained. One of the most common questions asked by the authors is how exactly to test for statistically significant differences supported by the Kruskal–Wallis test result. To check which groups differ, the Dunn or Dunn test with Bonferroni correction is usually used. Alternatively, Mann–Whitney tests with Bonferroni correction can be performed.3, 4 It is important to remember to also take into account an additional descriptive statistic such as the median. If multiple comparisons are referred to, they should be taken into account before the study begins in the design phase. Both the European Medicines Agency and the US Food and Drug Administration have described different ways concerning how primary and multiple secondary endpoints can be tested while maintaining overall alpha levels.5 Another aspect to consider concerns the false discovery rate (FDR). In a situation where many multiple comparisons are carried out, it is worth considering the number of incorrectly rejected null hypotheses. This refers to the number of falsely significant results in relation to the total number of all results that were found to be statistically significant. By calculating this ratio, the so-called level-q (q-value) can be calculated. Its interpretation is that a particular result considered statistically significant is in fact statistically insignificant. One of the main methods of controlling the FDR indicator is the Benjamini–Hochberg procedure.6 Conservative test, taking correction for the number of comparisons made. Used when the number of comparisons is small. Less prone to Type I errors than the Scheffe test. More rigorous than the Tukey test. The groups being compared must be equal It increases the likelihood of a Type II error. Weak statistical power. The corrected alpha value is often less than required The main outcome of a quantitative study should be the effect size. Unfortunately, this is still an underestimated parameter rarely calculated by authors these days. Compared to the p-value determining the statistical significance of the results, the effect size is a statistic indicating the strength of the phenomenon. In situations where, for example, there are small differences, but statistically significant, giving a p-value alone says little. Through the effect size, it is possible to know the practical meaning of the result obtained. It allows the results of studies to be compared in meta-analyses. Among other things, it is possible to calculate a summary estimate of the effect size, as well as to examine differences in effects between studies. Note that unlike significance tests, the effect size does not depend on the sample size. Unfortunately, it is common in science for authors to over-interpret the results obtained, which, for example, have to do with a very large sample size. After analysing the size of the effect, it may then turn out to be very small. Statistical significance does not automatically mean that this is a highly significant, practical result. It is important to remember to distinguish between statistical significance and practical significance.9 For this reason, it is recommended that the authors calculate the effect size so that the practical significance of the results obtained can be known. The most commonly calculated and well-known measures of effect size include odds ratio (2 × 2 table, logistic regression), Pearson's r (linear correlation), Spearman's ρ (rank correlation), r2 (simple linear regression) and adjusted r2 (multiple linear regression). Table 2 contains additional, namely less frequently calculated effect size measures by the authors, that is for the listed statistical tests. The minimum clinically important difference (MCID) should also be borne in mind. Statistical significance does not always show a relationship with clinical relevance. For this reason, it is important to try to determine whether the observed difference, supported by an appropriate measure of effect size, is noticeable to patients. It is a very helpful tool for planning research as well as calculating sample sizes. Based on the effect size and MCID, clinicians are able to make an evidence-based decision related to the care of a specific patient.10 0.2—small effect 0.5—medium effect 0.8—large effect 0.1—small effect 0.3—medium effect 0.5—large effect 0.01—small effect 0.06—medium effect 0.14—large effect 0.1–0.3—small effect 0.3–0.5—moderate effect >0.5—large effect In conclusion, the correct selection of an appropriate post-hoc test and the calculation of appropriate effect size measures by the authors allow for a more complete, reliable and practical demonstration of the research results obtained, which may consequently have an indirect impact on our quality of life. None. The author declares no conflict of interest.