Biomarker Failures

The quest for biomarkers has been a highly prolific, exciting field of research. Despite major promises, however, that biomarkers can improve diagnosis, prognosis, prediction, overall management, and eventually the health outcomes of single people and many different populations, the yield of successful biomarkers with unquestionably favorable health impacts has been extremely limited to date. In fact, biomarker failures are rather the rule. I try to classify common failures into some key recognizable types. The hope is that this classification may help in understanding what goes wrong in each case and offer some lessons on how we could try to avoid similar problems in the future. Table 1 highlights a number of aspects that need to be fulfilled in the translational history of a successful biomarker and shows 1 example of each type of failure. Indicated are the translational steps where the biomarker did poorly (−) or did well (+). Indicated are the translational steps where the biomarker did poorly (−) or did well (+). A type A failure (or clinical reversal) occurs when a widely used biomarker that has already been implemented in clinical practice is shown to be largely useless—or even harmful—and therefore needs to be abandoned. A typical example is prostate-specific antigen (PSA),4 in which after decades of debate and uncertainty, the recent recommendation has settled against routine screening with this marker (1). Some biomarkers, like PSA, were introduced into clinical practice at a time when clinical research in biomarkers and the respective applied statistical tools were still rudimentary. Many tests and interventions were adopted with standards that would be unacceptable today. It is not easy to understand why PSA and not others among the hundreds and thousands of proposed biomarkers moved so promptly into such wide clinical use. Perhaps its spread was facilitated because the test was considered relatively accurate and standardizable and because assays were also simple and easy to use across laboratories. Predictive discrimination was actually poor, however, and the test lacked a demonstration of actionable reclassification into risk categories in which the optimal managements would be different. Overdiagnosis rates were as high as 17%–50%, and treatment could produce serious harms. Extensive research was eventually performed, often from teams that explicitly wanted to show that PSA was a good or bad idea, and even large-scale evidence was obtained. But randomized trials failed to show any clear benefits. The introduction of the test into practice did not seriously take into account the possibility of harms from the use of a poor screening test. Somehow, early in the process indispensable pieces of necessary evidence were omitted along the path of introducing the test into practice, and one had to revisit these pieces of the evidence after the test had already become deeply entrenched in healthcare. I dare guess that the several other routine diagnostic tests that were introduced in the same era as PSA, or even earlier, probably have as miserable a performance record as PSA and may have to be abandoned sooner or later (2). The problem, however, is that once a test is introduced into practice, it is very difficult to remove it, even if the accumulated evidence seems convincing. Often the test continues to be used because of other reasons, such as conflicts of interest of specialist practitioners. For example, it is possible that PSA has survived for so long despite the evidence simply because it has been a great enhancer of clientele volume in urology practices. A type B failure (or validation failure) occurs when a biomarker shows great promise in one or more early studies, the claims are later found to be wrong or exaggerated, and the biomarker is eventually never implemented into clinical practice. An example is the early proposed discovery of proteomic patterns for ovarian cancer. The corresponding publication in The Lancet in 2002 (3) remains the most-cited proteomics research paper ever published, and it was likely instrumental in promoting the growth of the wider proteomics research agenda. The results suggested strong (practically perfect) discriminating ability and actionable reclassification that could change the screening practice and management of a lethal disease; however, it was the demonstration of a lack of analytical validity that killed this promise (4). Problems of analytical validity have been tackled to some extent since then, but they remain a limiting step in clinical proteomics, along with the difficulty in standardization (5). There has also been limited attention to the need to carefully validate results in external samples and to obtain large-scale evidence before testing and making claims about clinical impact (6). Because these problems are now recognized, there are signs that efforts are being made to address these deficiencies. One hopes that some proteomic biomarkers may have a better translation fate in the future (7). A type C failure (or nonoptimized clinical translation) occurs when a biomarker shows some genuine promise in one or a few early studies but this result is not followed up systematically toward clinical implementation. An example is gene expression profiles for cancer outcomes (8). Hundreds of such profiles have been proposed in the literature. If these results can be taken at face value, their discrimination is often very good, and their use has the potential to lead to modification of the management plans for some major cancers. Very few signatures have moved forward toward clinical development (9), however, and it is even unlikely that these promoted signatures are the best that have been developed to date. Although analytical validity and the use of proper statistical methods in the development phase are often properly addressed in this field (8), most studies remain of modest or even small sample size, and selection for further translation into clinical use is not necessarily driven by rigorous comparative evidence for the relative performance, merits, ease of use, and implementation of different proposed signatures. It is possible that interest by commercial sponsors is the main driving force toward translation and implementation. Sponsors may focus on one specific signature for which they have proprietary rights rather than take an overview of the entire evidence and select the best. A type D failure (or promotion despite nonpromising evidence) occurs when a biomarker shows no or little promise, but nevertheless is enthusiastically promoted for widespread clinical or population use. An example is the massive wave of personal genomics with genetic prediction models that use limited genetic information. Under the guise of recreational genomics, a number of companies have widely circulated genomic tests that can highlight a number of common genetic variants that have shown nominally statistically significant associations with various disease phenotypes. Analytical validity is exemplary in this field, because genotyping measurement error is minimal and standardization is feasible. Evidence for the associations has been generated from extremely well-conducted large studies and consortia with excellent quality control measures. There is little doubt about the statistical significance of these associations, but at the same time it is clear that, for the large majority of these diseases, the incremental information conferred by the identified common genetic variants is negligible (10, 11). Claims for large net reclassification index (NRI) improvements in the face of negligible improvements in discrimination are probably simply proofs of the instability of measures like NRI when there is flexibility in the selection of risk thresholds and analysis options (12). It can even be proved that for most diseases, even with many more common variants being identified, genetic predictive models are unlikely to be informative enough to be worth incorporating into clinical practice (13, 14). Certainly, there must be exceptions to this rule, and they are worth pursuing; however, the vogue about universal use of genetic prediction models cannot be justified on the basis of the current evidence. I have not included fraud in the categories of failures. Even if many of the failures entail conflicts of interest and spreading misinformation, such practices do not constitute fraudulent situations. An example of the spreading of misinformation is contained in the recent series of retracted papers by Potti [for a review, see (15)], in which the fascinating results promoting the use of molecular signatures in several clinical trials were irreproducible. I estimate that fraud is exceptionally uncommon in science, despite the vast attention that it receives in the scientific and public press when a case is discovered. Such examples, however, offer a lesson by suggesting that the principles and practices of reproducible research are worth introducing into biomarker research (16)—at a minimum when tests advance beyond mere scientific curiosity and move toward clinical testing for implementation in practice (17). Biomarker failures are unfortunate, but we can learn from them and apply them in future efforts. A main take-home message is that skipping any of the steps listed in Table 1 and cutting corners to expedite translation may not be a good idea. Development of biomarkers that work will take concerted efforts, and the necessary steps in the translation process need to be respected. Given that a few hundred thousand studies on biomarkers have already been performed and tens of thousands more are conducted every year, yet only a handful of biomarkers are used reliably in practice, there is no impetus to just publish more papers on biomarkers. We need to improve on our systematic evaluation of the evidence and on promoting—from each translational step to the next—the biomarkers that have the best evidence and can be the eventual winners. That will require a coordination of the wider biomarker research agenda. Such coordination may also benefit more from the fostering of international multiteam collaborations rather than from the fragmented efforts of small, opportunistic studies. prostate-specific antigen net reclassification index.