Learning to Live with Sampling Variability: Expected Replicability in Partial Correlation Networks

The topic of replicability has recently captivated the emerging field of networkpsychometrics. Although methodological practice (e.g., p-hacking) has been identified as aroot cause of unreliable research findings in psychological science, the statistical modelitself has come under attack in the partial correlation network literature. In a motivatingexample, I first describe how sampling variability inherent to partial correlations can merelygive the appearance of unreliability. For example, when going from zero-order to partialcorrelations there is necessarily more sampling variability that translates into reducedstatistical power. I then introduce novel methodology for deriving expected networkreplicability (ENR), wherein replication is modeled with the Poisson-binomial distribution.This analytic solution can be used with the Pearson, Spearman, Kendall, and polychoricpartial correlation coefficient. I first employed the method to estimate ENR for a variety ofdatasets from the network literature. Here it was determined that partial correlationnetworks do not have inherent limitations, given current estimates of replicability wereconsistent with ENR. I then highlighted sources that can reduce replicability, that is, whengoing from continuous to ordinal data with few categories and employing a multiplecomparisons correction. To address these challenges, I described a strategy for using theproposed method to plan for network replication. I end with recommendations that includethe importance of the network literature repositioning itself with gold-standard approachesfor assessing replication, including explicit consideration of type I and type II error rates.The method for computing ENR is implemented in the R package GGMnonreg.