Some mathematical reasons why meta-analyses of nonpharmaceutical interventions like facemasks could be inaccurate

The COVID-19 pandemic highlighted the need for non-pharmaceutical interventions (NPI) to mitigate hospitalization and death prior to vaccine availability. However, the precise impact of those mitigations have remained controversial. In particular, the Cochrane Review’s recent update on the effectiveness of NPI for respiratory infections was inconclusive, especially for facemasks and COVID-19, versus a number of similarly respected studies that showed efficacy and effectiveness. Here we show that the inconclusive results could have resulted from statistical analysis not suited to the non-linear mode of action of NPI. The Cochrane Review uses robust linear regression models to combine randomized controlled trial (RCT) data in their meta-analyses. Even basic models, however, show that NPI impact transmission in a non-linear way. We use a simplified, mechanistic, dynamic differential equation-based model to evaluate an NPI, facemasks, in a virtual RCT. The simulation is intended to illustrate the non-linear nature of the NPI impacts only and not intended to be a detailed replication of an actual RCT. NPI like facemasks have a variable impact on infection risk as a function of time, percentage of use, and basic reproductive number of the pathogen. Importantly, if the design, duration, and statistical power of the RCT are either resource limited or chosen poorly, the study will show no statistically significant effect, even when the NPI are benefitting the population exactly as predicted. Moreover, if the NPI are widely deployed and significantly reduce the effective reproductive number of the pathogen, a basic assumption of the RCT is violated, because the NPI are affecting the control group as well as the intervention group. Meta-analyses need to account for the facts that NPI effects are non-linear and not constant, and could be benefitting the control group through source control, all of which can produce non-significant risk ratios in an RCT, even though the NPI are performing exactly as designed. • Non-pharmaceutical intervention efficacy against respiratory pathogens is still controversial. • During the SARS-CoV-2 pandemic, some randomized controlled trials of NPI were inconclusive. • Meta-analyses of RCTs do not account for the non-linear mode of action of NPIs. • By impacting both arms of a study via source control, NPI can also violate an RCTs basic premise. • For these and other reasons, RCTs can under-estimate the true impact of NPI.