Sample Sizes for Magnitude-Based Inferences about Clinical, Practical or Mechanistic Significance

PURPOSE: The traditional method of sample-size estimation based on statistical significance is not appropriate for a study designed to make an inference about real-world significance, which requires interpretation of magnitude of an outcome. I present here two new methods for estimating sample size for such studies, based on (a) acceptable error rates for a clinical or practical decision arising from the study and (b) adequate precision for a mechanism-related inference. METHODS: For (a) I devised two new types of error: deciding to use an effect that is actually harmful (a Type 1 clinical error), and deciding not to use an effect that is actually beneficial (a Type 2 clinical error). I then constructed a spreadsheet to calculate sample sizes for chosen values of Type 1 and 2 errors, for chosen smallest beneficial and harmful values of various outcome statistics and designs (changes or differences in means in controlled trials or cross-sectional studies, correlations in cross-sectional studies, relative risks in cohort studies, and odds ratios in case-control studies), and for chosen values of other design-specific statistics (error of measurement, between-subject standard deviation, proportion of subjects in each group, and incidence of disease or prevalence of exposure). The calculations are based on the usual assumption of normality of the sampling distribution of the outcome statistic. For (b) I reasoned that precision is adequate when the uncertainty in the estimate of an outcome statistic (represented by its confidence interval) does not extend into values that are substantial in both a positive and a negative sense when the sample value of the statistic is zero or null. Sample sizes are then derived from the spreadsheet by choosing equal Type 1 and 2 clinical errors (e.g., 5% for a 90% confidence interval). The sample sizes for both methods can be compared with those based on the traditional method, included in the spreadsheet. Also included are confidence limits and quantitative and qualitative chances of benefit and harm for the “decision value” and any other values of the outcome statistic. RESULTS: Sample sizes for Type 1 and 2 clinical errors of 1% and 20% are ∼10% smaller than those for adequate precision with a 90% confidence interval, which in turn are only one-third of traditional sample sizes (for Type I and II statistical errors of 5% and 20%). Confidence limits and clinical chances provided by the spreadsheet are fully consistent with Type 1 and 2 clinical errors. CONCLUSION: Researchers can now justify and use sample sizes for studies aimed at making inferences about magnitudes. The sample sizes can be much smaller than those based on statistical significance.