Methodological Studies of Systematic Reviews: Is There Publication Bias?

Khan et al¹describe the results of a systematic review of published randomized trials of antiestrogen therapy in subfertile men with oligospermia. The odds ratio (OR) estimated from the 7 trials of "low" methodological quality (OR, 2.6; 95% confidence interval [CI], 1.2-5.2) suggested a greater treatment effect than the OR estimated from the 2 "high-quality" studies (OR, 0.5; 95% CI, 0.2-1.5). The authors conclude, "It is clear that low-quality studies are more likely to produce estimates of treatment effect that are exaggerated." How "clear" is it really? First, the 95% CI around the 2 ORs overlap, indicating that the difference between the ORs did not reach conventional statistical significance. Second, the difference seems to rely almost entirely on the results of 1 high-quality trial,²a study of 23 men that found a low OR of 0.1 (95% CI, 0.004-1.8). Caution seems prudent if the results of a systematic