A case study on applying p-curve analysis to clinical trial data

Background Publication bias may occur when smaller studies reporting statistical significance ( p < 0.05) are more likely to be published than smaller studies that do not report statistical significance. One way to detect publication bias may therefore be to detect a skewed distribution of reported p-values in a set of published articles. Aim To showcase the complexity and uncertainty of p-curve analysis as a method to detect publication bias. Methods We re-analyzed a meta-analysis summarizing randomized controlled trials of treatment for depression, using p-curve analysis. One significant p-value per study was selected for the main analysis, with alternative significant p-values allocated to a robustness analysis, which we called “robust p-curve” . We used the “p-curve web application” (https://www.p-curve.com/app4/). Results Our dataset included 36 test statistics from 29 of 41 (70.73%) studies in the meta-analysis. The corresponding p-curve was neither significantly right-skewed (z = -1.03, p = .15) nor significantly flatter than a curve powered at 33% (z = -1.02, p = .15); it was, therefore, inconclusive. The robust p-curve was similar, though its right-skewness test was not statistically significant either (z = -1.61, p = .05; flatness z = − .55, p = .29). Both p-curves suggested that the average power of the included studies was low [Estimate: 14%, 90%CI (5%, 48%); Estimate: 22%, 90%CI (5%, 57%)]. Conclusions Based on p-curve analyses, we could not rule out or rule in publication bias. One of the reasons may be that the sample size of the studies included was too small. P-curve analysis complements, rather than substitutes for, other methods for investigating publication bias. Our study highlights the assumptions and uncertainty that researchers must deal with and reflect on when analyzing potential publication bias.