Commentary: Grading the credibility of molecular evidence for complex diseases

Dissecting the aetiology of complex diseases has been a great challenge for biomedical research, including epidemiology. Several thinkers,1–4 including Buchanan et al.5 recently, have focused on the unquestionable difficulties of this ambitious enterprise and the great obstacles encountered in the way. Some of them have ended up with a futility outlook. Over more than a decade, the debate has ranged wild on whether epidemiology has reached its limits,6 is either dead or in a vegetative state, should call it a day, and whether ‘it is time for scientists to re-think the quest’ and realize that ‘base metal cannot be turned to gold’.5 In this commentary, I will not argue whether the challenges for attaining evidence are formidable in the current, molecular era. I will certainly not propose that the ‘hunger of the paying public for easy answers and promises’5 should be superficially satisfied. However, I will argue that not only the public, but scientists also, have been starving until now for some tractable knowledge. I will also argue that despite all shortcomings, molecular evidence, with molecular epidemiology as a centre piece, does have a future. I will try to demonstrate that reasonable progress can be achieved; that progress will require scientific humility and the realization that many postulated research findings have been false; that false discoveries will continue to be very common; and that we need to adapt from the concept of solid knowledge taken for granted to the concept of tentative information that should be replicated and scrutinized. Finally, I will propose a hierarchy for grading the credibility of molecular evidence in complex diseases that emerges under the current circumstances. Critics of molecular evidence contrast the difficulties that arise in trying to understand complex diseases against a glorious past of biomedicine. As Buchanan et al.5 claim, the failure of current research is in ‘stark contrast to prior decades of success in which both epidemiology and human genetics uncovered major causal risk factors … and lent [their] fields … a well-earned authority.’ I greatly respect past successes of these fields, even more so, since they were achieved with limited means and crude methods. But epidemiology and genetics are not dead or finished disciplines—they are just starting now. Biomedicine until the mid-20th century had been to a large extent a compilation of unfounded beliefs and often dangerous practices, variously infiltrated by vague dogmatic theories derived from the equally or even more immature and overconfident physicochemical sciences. Who would like to defend an era before the mere existence of randomized trials and when even demonstrating major risk factors such as smoking for lung cancer met with enormous resistance from the establishment,7 while academic stupidities circulated at large? As we get closer to our times, we may have more resistance in admitting the limitations of our professions and scientific disciplines. Nevertheless, we have increasing empirical evidence that the performance of biomedical research has not been that spectacular to date—in any front, be it basic science, preclinical science and epidemiology, or clinical research. At a minimum, there is certainly much room for improvement. Among 101 publications in major basic science journals between 1979 and 1983 that made clear promises for resulting in major clinical applications, only one materialized in a widespread clinical application in the subsequent 20–25 years.8 The translation of basic science in less prestigious journals and among the vast majority of basic research that does not even imagine being translated is unknown, but it is likely to be even less efficient.9,10 Much of the ‘basic’ science research to date is confined to ‘focused’ observations recycled within esoteric circles of similarly oriented sub-sub-specialists without any material consequence whatsoever. When it comes to epidemiological investigation and clinical research, the situation is not necessarily better. There is increasing concern that the quality of both epidemiological investigations11 and clinical research, including clinical trials,12–14 has been quite poor until now. Exceptions of brilliant and well-designed studies only reinforce the rule. It is not surprising then that the findings of epidemiological and clinical research are very often refuted. In an evaluation15 of the 45 most-cited (over 1000 citations each) articles finding significant effects for interventions published between 1990 and 2003, five out of six non-randomized studies had already been either clearly refuted by subsequent research or found to have proposed seemingly exaggerated effects by 2004. Even among randomized trials, a quarter had already been refuted or found to be exaggerated. These data pertain to highly-visible research published in the best, most competitive journals. Less visible research sometimes has been so bad that refutation is not even an option. A scientific claim can be refuted only when it is coherent enough to allow refutation. Much research in the past has been performed with bad methods, for the wrong reasons, under heavy conflicts of interest, or with combinations of these flaws. Even randomized clinical trials suffer from publication bias,16 time lag bias,17 and extensive selective reporting.18 As Doug Altman19 pointed out several years ago, poor medical research is a scandal. Defenders of the ‘macroscopic’ epidemiology of the pre-molecular era might think of several reasons why to prefer traditional risk factors over mysterious molecular risk factors. Postulated risk factors like obesity, consumption of fruits and vegetables, or consumption of coffee are readily tangible. One can communicate to every lay person that obesity is bad, eating fruit and vegetables is good, and drinking coffee is bad, or good, or does not matter. Anyone can understand what obesity, a peach, or a cup of coffee is. Conversely, most of the genes and gene variants unearthed by molecular research (even when they have a name that is not created simply by a cryptic string of vowels, consonants, and numbers) remain a mystery to the layperson and sometimes also to the scientist-expert. The other postulated problem with most candidate molecular risk factors is that usually one cannot modify them—to date at least. One can advise people to lose weight, eat cucumbers, and drink black coffee in moderation. One cannot tell them to change a single nucleotide polymorphism (SNP) that they already carry in their genome. Last, one might argue that if 200 molecular factors act cumulatively, additively, synergistically, or competitively along with 200 multifarious environmental exposures to generate obesity, why not just focus on the final visible, tangible macroscopic risk factor, i.e. plain body fat, rather than try to chase 400 molecular ones? The reason why we have to go after these 400 risk factors is that we really have no other good choice. We have tried our best with macroscopic risk factors and this is what we got. A few years ago, the obesity epidemic was estimated to account for an extra 300 000 deaths in the USA,20 while recently it was claimed that there is hardly any independent effect on mortality, and probably more deaths are caused from underweight than from overweight.21 Some studies had found that fruit and vegetables reduce the breast cancer risk 10-fold,22 while recent large cohorts find no effect at all.23 Coffee consumption was one of the first major ‘risk factors’ to be proposed and then retracted as a causative factor for pancreatic cancer.24,25 Buchanan et al.5 provide an excellent list of irreproducible findings of the past. Furthermore, even in the instances where reproducibility has been relatively good, the premise that these factors can be easily modified is spuriously optimistic. Finally, I do not believe that the ability of tabloids to accurately name or describe a risk factor should be taken as scientific proof for its importance. I think it simply means that many low-yield ‘macroscopics’ should be left to tabloids and epidemiology should proceed to the molecular era. If macroscopic risk factors are indeed the result of the interplay of hundreds of other more proximal risk factors then it should not be surprising that our results with traditional risk factors have been so unstable to replicate. Large heterogeneity is only to be expected. Results obtained with such risk factors may be difficult, if not impossible, to generalize to other patients and populations. Traditional risk factors are very crude composites of very heterogeneous risk quanta each of which may have a small impact on the macroscopic risk factor and on the final outcomes of interest. Obesity may be possible to describe eventually as a composite of 3000 different sub-types, and fruit consumption may be an extremely complicated biological phenomenon to which we may be doing gross injustice by simply measuring rations consumed per day. Molecular ‘progress’ may be difficult, if common complex diseases are the result of chance or if these diseases largely have a highly ‘private’ genetic epidemiology. For example, if each case of a lethal disease is caused by a new mutation and each new case represents still a new genetic variant that has not been previously encountered, we can keep cataloguing this knowledge but this will not be of prognostic value for the general population. Even then however, we might gain some interesting scientific knowledge (as opposed to practical, usable knowledge) from the mere evolving catalogue. Chance and private genetic epidemiology may have their share indeed for the aetiology of complex diseases. In fact one might consider ‘chance’ as the private environmental epidemiology, equivalent of and working with or against the private genetic epidemiology. However, probably for most common and important complex diseases, there is a sizeable component of their aetiology that can be ascribed to measurable and reproducible risk factors, be it genetic or environmental, or both. Evolving and improving multivariate predictive models may never reach a coefficient of determination, R2, of 1.00. However, it is reasonable to expect that we can improve these coefficients of determination for most complex diseases by starting to incorporate molecular factors. Examining how much we can improve them for each disease is actually a very interesting scientific question on its own. Current predictive knowledge is probably largely over-fit, biased, unchallenged, and non-validated. Most claims about strong predictors and large R2 for many diseases are exaggerated and would not stand the test of large-scale independent validation in unbiased assessments.26 The threat for poor validation and exaggerated expectations for diagnostic accuracy and prognostic performance of multivariate models may be even greater in the molecular era.27–30 We may need to take a step back and acknowledge that we know less than we think we do. For many diseases, we may have to acknowledge that we know practically nothing that can stand much testing; anything we learn from now on would be a plus, even if biology and aetiology prove to be private to some extent. Molecular medicine and molecular epidemiology are characterized by a rapid, exponential increase in the number of putative risk factors that can be measured. A couple of decades ago, the usual practice had been to focus entirely on a single candidate risk factor and at most consider a handful of other measured variables as adjustments to account for confounding. Ten years ago, measurements in the most complicated studies could master only a few dozens of candidate risk factors. Single experiments can currently measure tens and hundreds of thousands of candidate risk factors at the same time. This is typically exemplified by the expanding applications of genomics, proteomics, and metabolomics. It translates to over a 100-fold increase per decade in the candidate risk factors targeted in a typical study. If the trend continues, studies with billions of candidate risk factors may become possible before my generation passes away! Even if the numbers of candidates reach a plateau soon, the number of potential candidates is already formidable. It is unlikely that many of the probed candidates are really important. Even if chance and private epidemiology do not account for a big proportion of the aetiology of common complex diseases, there is probably a finite number of molecular risk factors that are operating for each disease. The smaller the average effect size for each of these risk factors, the larger their likely number. As shown in Table 1, we have evidence from studies to date that effect sizes in molecular epidemiology31–38 may cover the same range as the effect sizes described in pre-molecular medicine and epidemiology. Small effect sizes are not peculiar to the molecular era. In fact, the overwhelming proportion of epidemiological associations ascertained or proposed in the past have been small effects. This applies also to experimental clinical designs. The typical examples are medical treatments: most established medical treatments to date have very modest relative risk reductions, in the range of 10–40%, i.e. small or very small effects. However, the demonstration of such effect sizes has been left to randomized experiments, when it comes to treatment efficacy. It has been questioned whether epidemiological studies can function as well in this range.39,40 There is thus some understandable agony as epidemiological methods are relied upon to try to capture and validate small or very small effects. Effect sizes in the pre-molecular era and in the molecular eraa RR: relative risk. a Presented examples reflect current state of knowledge and are subject to possible refutation in the future; for small and very small effect sizes, it is uncertain whether these risk factors are true, even when evidence is based on large sample sizes from several studies. Effect sizes in the pre-molecular era and in the molecular eraa RR: relative risk. a Presented examples reflect current state of knowledge and are subject to possible refutation in the future; for small and very small effect sizes, it is uncertain whether these risk factors are true, even when evidence is based on large sample sizes from several studies. Regardless of the agony, since small and very small effects were being pursued already by macroscopic epidemiology, the challenge is not new. Moreover, I see no reason why someone should worry because epidemiologists can measure now a lot of variables. We had good reason to complain in the past that we could only measure a few variables; that our measurements were crude; and that most of the essential players remained unmeasured and unapproachable, collectively buried under residual confounding. Now we can measure a lot of things and the ability to measure keeps improving. This is good news. There are also other issues to consider this exponential improvement of measurement agility. The availability of tons of complex data may lead to greater temptation to perform data dredging and selectively the most but biased, This is certainly a threat and it is probably currently as new and are their Conversely, the trend may when these scientific When tons of data are the and need of to data for results may A of 000 is likely to many thousands of putative risk factors and even with a couple of of experiments, several candidates may for A is with the problem of still many risk factors on to data this against the of a decade to the data on one or candidate risk factors and of would be to go wild on data to up with some but probably effect to In the data is likely to be less if 000 candidates are all at rather than if 000 candidates were to on the over 000 000 We need to get to the of so get there as as This new a scientists were to and information that they was of such information created agony, and public We should now that most of the biomedical information that is likely to be and will be of solid we should get to the of tentative single in the molecular no how and is probably more likely to be refuted rather than This does not we should these We should just them for what they tentative some of a small may eventually reach of while much will be refuted. this one could consider a grading of credibility of biomedical information in the molecular era. It can be that in the of large effect sizes are probably more than smaller effect However, a would be the ability to molecular findings and in studies. Table the range of credibility of biomedical information to the effect size and the extent of of the credibility of research findings to effect size and extent of credibility of research findings to effect size and extent of A very small effect size risk even if found significant in a sizeable is very unlikely to be in a research of where thousands and of such findings are likely to up by Even with extensive many it would still be more likely for this finding to be false rather than true, it is not only by several studies but also by other independent of Small and effect sizes may reach a credibility of or if such extensive is while for very large effects credibility may reach up to or at Nevertheless, no candidate risk factor can be for to be The vast majority of proposed risk factors will continue to be at the of the credibility We need to that at any time or more of our tentative information in our other of or these will be this may with my but I think that or more of our tentative information is already false Regardless of whether I or not about the molecular era it clear that we need to with humility and these credibility This has major on how research findings are and eventually It also has major on why and so important. We cannot on single studies. Single studies are and they are important to respect and but they will not provide the final We need and of all studies that are in any and of even more and are is not but its credibility to be and In this of working on the same with information may become the These would not and brilliant new research but research findings are by would then be possible to test and validate or the of other working on the same an is already being in several and a of has recently been created in human in an to share on how to and such of may also to keep the molecular evidence of their the one might consider a grading of molecular evidence, as shown in Table The grading five effect and of the evidence, from biological and grading of credibility in molecular evidence grading of credibility in molecular evidence I have already the first effect size and of the One should some effect In the of the effect sizes may actually simply measure nothing but the average operating in a scientific As shown under of larger effect sizes simply that more is of the good to be the data being we actually have a chance of measuring in molecular research. the in the very important. In the of strong even strong effects that have been several studies may need to be fields of scientific may be in this way. The problem is that for most currently research, we are very limited or no information by which to whether has been or in epidemiological studies is esoteric and the to improve in the and of research will the it is to results as if at some is until credibility is also important to but we also need more data on how this should be It is easy to claims about biological Much of that may be but there is no to measure However, we do to data on molecular associations from other of We need more empirical data in to understand what they i.e. how much they should change the credibility of some molecular epidemiological For example, how should associations be when an is in a or a much should the credibility improve when a an effect of this polymorphism on much should data on the credibility of an epidemiological some evidence that biological data are it is that different experimental may different The increase or in credibility based on biological and data will continue to be a choice. However, we can now and the of biological the we can some empirical evidence on what each of biology It will not be but it will be that we can this to the of and more than the by biological may become of The to consider is the of a molecular research at most research findings have no major clinical or public We should be to acknowledge this a risk factor that for of the risk of a disease is a and most are likely to be of this However, this does not that for this risk factor should be to the general It is that most molecular discoveries will continue to have limited for public and clinical This humility should not be should the public be with easy A the would that the final for cancer is many per should and the public that scientific progress is smaller and these should be for what they In we need to learn from our and We may never metal to but we still have a lot of things to learn from the of in molecular epidemiology.