Incidence Proportions From Random-Effects Meta-Analyses of Rare Events With Large Cluster Heterogeneity, Including Sensitivity to the Normal Distribution Assumption

Our narrative review examines statistical methods for estimating incidence proportions of rare events (≤5%) with large heterogeneity among clusters. One example dataset is a meta-analysis of incidences of neurological complications after regional anesthesia procedures. The clusters (i.e., studies) are nested within groups, whether they use ultrasound guidance or not. The other example dataset consists of ratings of faculty anesthesiologists (i.e., the clusters represent the raters). We review weighted mean proportions among clusters (i.e., marginal estimates). We also review incidence proportions for the median cluster (i.e., conditional estimates). Because marginal estimates are larger than conditional estimates, we recommend knowing which one is being reported and, for each application, consider which would be preferred. For both example datasets, the logits of the observed incidences for each cluster followed normal distributions within groups, satisfying statistical assumptions of random-effects logistic regression. Accurate confidence interval coverage for conditional and marginal estimates for incidence proportions can therefore be obtained using random-effects logistic regression. We recommend this method and provide Stata code. Probit regression depends on probits following a normal distribution, which was used for one of the two example datasets. Calculating incidence proportions for each study individually, along with variance estimates, and then pooling, yields different results (e.g., a 100% relative error in the estimated incidence). We review that the two-step methods for incidence proportions with the closest to nominal confidence interval coverages are those performed with arcsine transformation or the Freeman-Tukey double arcsine, with the inverse calculated based on the harmonic mean sample size. Although these methods are suitable for forest plots, we recommend against using them for primary inference due to their inferior performance compared to random-effects logistic regression. Hospital management reports using anesthesia data can employ methods that misuse data, neglecting the heterogeneity among clusters (e.g., generalized estimating equations). For example, consider hospital reports of postoperative infections among patients undergoing surgery, which are pooled by surgeons or surgical procedures. Even when there is an absence of interest in the variability of incidences within clusters (e.g., among obstetricians) or even consideration of the clusters (e.g., individual procedures), fixed effects methods (e.g., simply pooling counts) have 95% confidence intervals with coverage <50%, their narrowness falsely suggesting precision. While such situations are generally obvious for meta-analyses of journal articles, often they are not so for managerial applications relevant to anesthesia. We recommend using random-effects models for these administrative and quality improvement reports on rare anesthesia events and large or unmeasured heterogeneity among clusters.