Acknowledge and fix the multiple testing problem

Vandenbroucke1 comments ‘Observational research that is started with equally high priors can be expected to perform as well as randomized trials.’ Empirical evidence is on the other side. It is not just hormone replacement therapy that has come a cropper. What about vitamins E and C, and selenium and multivitamins, to name but a few?2–4 Maziak5 points to what appears to be a general problem: claims coming from observational studies generally fail to replicate. See also Ioannidis6 and Boffetta et al.7 One small step forward would be to acknowledge that multiple testing is likely part of the problem. How can you ask tens to hundreds of questions and not expect false positives? Those that argue for no correction for multiple testing8,9 leave me not persuaded. One purpose of a scientific paradigm is to lead to good practice and empirically verified results. Why continue a data analysis strategy that is wrong 80–90% of the time? Multiple testing should be fixed before someone says, why fund any of this research?