The Influence of Display and Statistical Factors on the Interpretation of Metaanalysis Results by Physicians

OBJECTIVE: The objective of this study was to determine the extent to which various factors affect the interpretation of metaanalytic results by physicians. STUDY DESIGN: A sample of 120 physicians, selected from The Royal College of Physicians and Surgeons of Canada (RCPSC), was randomly assigned to 1 of 6 groups (n = 20) created from a combination of 3 summary measures and 2 levels of disease severity. The intervention consisted of a written scenario and 4 individual displays of metaanalyses (M-A), each followed by questions related to the interpretation of results of M-A. Two final questions examined statistical familiarity/proficiency with the summary measures used. DATA ANALYSIS: Analyses of variance examined main effects and interactions among 4 factors: summary measure, disease severity, effect size, and statistical consistency of the studies comprising the metaanalysis. Two outcomes were examined: interpretation of the treatment effect and confidence in the interpretation of the treatment effect. PRINCIPAL FINDINGS: Physicians were more likely to favor treatment when the results of the primary randomized, controlled trials (RCTs) were statistically homogeneous (P = 0.001) and when the overall effect size was large (P = 0.001). Also, physicians were more likely to be confident when the results were homogeneous (P = 0.001) and when effect size was large (P = 0.000). Interactions also revealed that the effect of statistical consistency of contributing to RCTs was greatest when data were presented as risk difference for treatment outcome (P = 0.026) and when effect size was small (P = 0.000). CONCLUSIONS: The interpretation of metaanalytic displays is influenced by the overall effect size of M-A, the statistical consistency of the contributing RCTs, and interactions of these factors with display factors.