Answers are grounded in this paper and cite their sources.
Analysis discusses barriers to clinical integration of omics-based tests, suggesting improvements for future applications.
“Why?” is the inevitable question. Why have so few biomarkers made it to everyday clinical care? Why, despite billions of dollars worldwide in omics-based research? We have been promised multiple breakthroughs, and numerous biomarker discoveries have been announced, but it is fair to say that, up to this day, clinical medicine has not gone through a radical change, despite all the investment of time, money, and the collaboration of thousands of study participants. Why has this omics-based enterprise resulted in so few clinically useful tests that improve health outcomes or contribute to healthcare efficiency? Sometimes things have gone badly wrong in the process of marker discovery and evaluation, to the extent that the things that went wrong risked undermining the credibility of biomarker research itself. Likely the most famous instance is the Duke University case, in which 3 gene expression–based tests were used in phase II clinical trials. The tests were supposed to be chemosensitivity tests, to guide therapy in patients with breast cancer and lung cancer. It turned out that the tests were based on shaky evidence and poor methods, and that their use in clinical trials could be called premature at best. Outside researchers argued that the omics-based tests did not work and were potentially endangering patient safety by incorrectly directing therapy (1). In 2010, National Cancer Institute (NCI)2 Director Harold Varmus received a letter from >30 scientists, voicing concerns about the tests used in the clinical trials at Duke University. In response, the Institute of Medicine (IOM) started a project to investigate the issues. This led to a report, published in 2012, titled, “Evolution of Translational Omics: Lessons Learned and the Path Forward” (1). IOM also formed a Committee on the Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials, and NCI organized a workshop on these issues. A working group then developed a checklist on the basis of the key principles in the IOM report and the results of the NCI workshop (2). A short version appeared in Nature last year, and a version with a longer explanation and elaboration was published in BMC Medicine (3). In these reports, omics refers to “the study of related sets of biological molecules in a comprehensive fashion.” An omics-based test is defined as “an assay composed of or derived from multiple molecular measurements and interpreted by a fully specified computational model to produce a clinically actionable result.” These documents were primarily written to support decisions about the use of omics-based tests in prospective clinical trials. The emphasis is strongly on treatment selection rules: combinations of markers that can be used to support therapy decisions, stratify clinical management, or otherwise guide therapy. This checklist can help in appraising the total body of evidence for an omics-based test: preanalytical requirements, analytical and clinical performance, and potential clinical utility. It also helps in understanding and positioning ethical, legal, and regulatory issues. It should come as no surprise that the list is quite long: it contains no fewer than 30 criteria, and the companion document for explanation and elaboration has 22 dense pages. Yet for those interested in the evaluation of novel markers, the list and the documents provide interesting reading material. It will be reassuring to laboratory professionals that the criteria pay attention to methods for the feasibility of obtaining samples, sample collection and processing, criteria for screening out inadequate or poor-quality samples, standard operating procedures used by the laboratories suggesting the test, required turnaround time, and technical artifacts. Statisticians and methodologists may be relieved to learn that there is mention of the “appropriateness of statistical methods” used in building the multimarker model (although in fairly general terms) and full specification of all cut-points and methods to assign confidence measures to predictions. An interesting addition is the invitation to “search public sources, including literature and citation databases, journal correspondence, and retraction notices, to determine whether any questions have been raised about the data or methods used to develop the predictor or assess its performance, and ensure that all questions have been adequately addressed.” This checklist not only can help in making decisions about trials, but also may be useful for systematically reviewing the body of evidence for novel tests, especially for multigene assays or multimarker models to guide treatment decisions. The committee stated that the “ultimate goal is to develop a more efficient and reliable process to move omics assays from promising research results to clinically useful tests that improve patient care and outcomes” (2). Is this set of criteria sufficient? Will it prevent events like those at Duke University and, more generally, curb the growing global disappointment with omics-based research, and lead to the accelerated introduction of novel tests that benefit us all? Certainly, this list will help, but more is needed. We believe there is more at stake than poor statistical methods, a limited understanding of preanalytical and analytical issues, and an absence of institutional structures to oversee research. The IOM report and the NCI documents firmly maintain the current paradigm, where marker research is neatly divided into a discovery phase, a test validation phase, and subsequent evaluations of the clinical utility of the resulting test. Yet, the selection of patients and samples used in the discovery phase—a population that often is hardly comparable to the intended use population—may be one additional reason for the disappointing performance in validation and evaluation phases. Like most of biomedical research these days, omics-based research is in need of external funding and fueled by high hopes, a combustible combination. There is selective and twisted reporting of study outcomes, and a failure to publish all studies, regardless of their findings. Prospective registration of all trials, not just those dealing with pharmaceuticals and other interventions, but also trials of omics-based tests and other markers, could definitely help here (4). Reporting guidelines, such as Standards for the Reporting of Diagnostic Accuracy Studies (STARD), can help in improving the completeness and transparency of study reports. When study findings are eventually submitted, many study authors find it hard to resist the temptation (all too human) to present their study results in a rosy light, to spin their findings, emphasizing the positive and the promise of their study in the conclusions, while minimizing problems and pitfalls (5). Here editors and reviewers should do a better job: they could identify spin in time, and invite study authors to attenuate their optimism when indicated. Optimists we must be, as patients and taxpayers, while waiting and hoping for the major breakthroughs. As laboratory professionals, as researchers and decision-makers, we must apply appropriate caution, close to an attitude of skeptical criticism. National Cancer Institute Institute of Medicine.
No takes yet. Share an insight, caveat, or question.
Patrick M. Bossuyt (2014) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: