Scientific Methods and the Reporting of Negative Results: Critically Important to Patient Safety

Vascular surgeons are no strangers to adopting new technology. After its introduction in 1991,1Parodi J.C. Palmaz J.C. Barone H.D. Transfemoral intraluminal graft implantation for abdominal aortic aneurysms.Ann Vasc Surg. 1991; 5: 491-499Abstract Full Text PDF PubMed Scopus (2972) Google Scholar endovascular aneurysm repair (EVAR) supplanted conventional open surgery in many jurisdictions as the dominant modality for treating aneurysmal disease.2Sethi R.K.V. Henry A.J. Hevelone N.D. Lipsitz S.R. Belkin M. Nguyen L.L. Impact of hospital market competition on endovascular aneurysm repair adoption and outcomes.J Vasc Surg. 2013; 58: 596-606Abstract Full Text Full Text PDF PubMed Scopus (45) Google Scholar, 3Chang D.C. Comparison of long-term survival after open vs endovascular repair of intact abdominal aortic aneurysm among medicare beneficiaries.JAMA. 2012; 307: 1621Crossref PubMed Scopus (100) Google Scholar The early adoption of this technology can be attributed to two equally important factors: one logical and one evidentiary. First, the concept behind EVAR (i.e. the exclusion of the diseased aortic wall from the circulation to prevent pressure-related expansion and rupture), is in accordance with our understanding of the pathophysiology of aneurysmal disease. Second, randomized controlled trials4de Bruin J.L. Vervloet M.G. Buimer M.G. Baas A.F. Prinssen M. Blankensteijn J.D. et al.Renal function 5 years after open and endovascular aortic aneurysm repair from a randomized trial.Br J Surg. 2013; 100: 1465-1470Crossref PubMed Scopus (45) Google Scholar, 5United Kingdom EVAR Trial Investigators Greenhalgh R.M. Brown L.C. Powell J.T. Thompson S.G. Epstein D. et al.Endovascular versus open repair of abdominal aortic aneurysm.N Engl J Med. 2010; 362: 1863-1871Crossref PubMed Scopus (1151) Google Scholar comparing EVAR with conventional surgery have demonstrated a rigorous, reproducible, and statistically significant perioperative benefit. But what is even more important about the introduction of endovascular techniques into modern vascular surgical practice is that iterative changes have occurred from the first generation devices, to make them safer and more durable. The successful evolution of EVAR technology has been the product of careful reviews of individual centre experience, pooled registry data, critical appraisal of the reasons for device failure, and the courage of leaders in our field to report negative results. This has also held true throughout the development of endovascular options for treating thoracoabdominal disease, where devices have evolved because centres have been open about the various modes of failure and those areas that required technical improvement. The Multilayer Stent (Cardiatis S.A. Limited Company, Isnes, Belgium) is a new technology, and evidence about its effectiveness is emerging in the peer-reviewed literature. Sultan et al. reported “aortic related” mortality rates of 0% at 30 days and 1-year survival rates of 87% in a population of 103 patients who demonstrated an average 0.57%/month sac volume expansion.6Sultan S. Sultan M. Hynes N. Early mid-term results of the first 103 cases of multilayer flow modulator stent done under indication for use in the management of thoracoabdominal aortic pathology from the independent global MFM registry.J Cardiovasc Surg (Torino). 2014; 55: 21-32PubMed Google Scholar Vaislic et al. reported their experience in 23 patients with a stable aneurysm diameter at 12 months and a 12-month all-cause mortality of 4%.7Vaislic C.D. Fabiani J.N. Chocron S. Robin J. Costache V.S. Villemot J.-P. et al.One-year outcomes following repair of thoracoabdominal aneurysms with the multilayer flow modulator: report from the STRATO trial.J Endovasc Ther Off J Int Soc Endovasc Spec. 2014; 21: 85-95Crossref PubMed Scopus (47) Google Scholar Ruffino et al. reported their multicentre experience in 54 patients with a 5.5% 1-year mortality and a decrease in aneurysm diameter of 11% by 12 months.8Ruffino M.A. Rabbia C. Italian Cardiatis Registry Investigators GroupEndovascular repair of peripheral and visceral aneurysms with the Cardiatis multilayer flow modulator: one-year results from the Italian Multicenter Registry.J Endovasc Ther Off J Int Soc Endovasc Spec. 2012; 19: 599-610Crossref PubMed Scopus (69) Google Scholar In another paper discussing the use of the Multilayer Stent outside of the manufacturer's instructions for use, all-cause mortality was reported to be 89%, but this was attributed to improper use of the device.9Sultan S. Hynes N. Sultan M. MFM CollaboratorsWhen not to implant the multilayer flow modulator: lessons learned from application outside the indications for use in patients with thoracoabdominal pathologies.J Endovasc Ther Off J Int Soc Endovasc Spec. 2014; 21: 96-112Crossref PubMed Scopus (40) Google Scholar However, in this months' issue of the European Journal of Vascular and Endovascular Surgery, Lowe et al. report the Manchester experience with the Multilayer Stent in 14 patients under the auspices of a UK pilot study.10Lowe C. Worthington A. Serracino-Inglott F. McCollum C. Multi-layer flow-modulating stents for thoraco-abdominal and peri-renal aneurysms: the UK pilot study.Eur J Vasc Endovasc Surg. 2016; 51: 225-231Abstract Full Text Full Text PDF Scopus (24) Google Scholar Unlike the preceding “positive” papers, Lowe et al. reported a 50% mortality rate (7/14) after 19 months follow-up and a 35% rate of reintervention, with most patients having continued sac expansion at 1 year having deployed the device according to the manufacturers' instructions.10Lowe C. Worthington A. Serracino-Inglott F. McCollum C. Multi-layer flow-modulating stents for thoraco-abdominal and peri-renal aneurysms: the UK pilot study.Eur J Vasc Endovasc Surg. 2016; 51: 225-231Abstract Full Text Full Text PDF Scopus (24) Google Scholar These results do not, therefore, demonstrate evidence of effective treatment of aneurysms using the Multilayer Stent and are in conflict with the previously published literature. The scientific method involves the systematic process of first asking a well-researched question, defining and testing a hypothesis and then analyzing the findings. When the findings do not match the hypothesis, these new “negative” data can be used to better inform a revised hypothesis. The step of critical analysis and conclusion forming is fundamental to the advancement of knowledge, as it allows the scientist to learn and respond to new information. However, should negative results go unpublished or un-presented, bias is introduced, the scientific method becomes perverted and no longer serves its fundamental purpose of expanding our body of knowledge. The absence of negative results (or reporting bias) in the literature is not a new phenomenon. A review of trials for new drugs approved between 1998 and 2000 by the Food and Drug Administration found that only 43% (394/909) of supporting studies were published within 5 years of their approval, suggesting there is a wealth of knowledge that never reaches the public domain.11Lee K. Bacchetti P. Sim I. Publication of clinical trials supporting successful new drug applications: a literature analysis.PLoS Med. 2008; 5: e191Crossref PubMed Scopus (173) Google Scholar In a recent simulation study looking at the effect of underreporting negative results in meta-analyses, Kicinski showed an increase in bias with the absence of negative results, which may be underestimated by the use of funnel plots to estimate publication bias.12Kicinski M. How does under-reporting of negative and inconclusive results affect the false-positive rate in meta-analysis? A simulation study.BMJ Open. 2014; 4: e004831Crossref Scopus (36) Google Scholar A recent study looking at the number of null results in the National Heart, Lung and Blood Institute trials database has shown an increase in the proportion of trials with “negative results” following the mandatory requirement for preregistration of randomized trials in clinicaltrials.gov. In fact, only 2/25 studies (8%) published after 2000 reported a “positive outcome,” compared with 17/30 studies (57%) published before 2000.13Kaplan RM, Irvin VL. Likelihood of null effects of large NHLBI clinical trials has increased over time. Garattini S, editor. PLoS One 2015;10(8):e0132382.Google Scholar There is no question that a failure to publish negative results will inappropriately influence the evolution of clinical practice and so it falls on all of us (as clinicians and consumers of the medical literature) to determine why this is happening. In this respect, the impact of industry sponsorship of trial research is one area to be explored. Although in some jurisdictions expenditure on research and development by governments has not been affected by the economic downturn, both the USA and the UK have reported trends toward an increasing reliance on industry funding of research.14OECD. OECD Report on GDP expenditure on research and development. Tableau Software. http://public.tableau.com/views/RDintensity/Dashboard2?:embed=y&:showVizHome=no&:host_url=http%3A%2F%2Fpublic.tableausoftware.com%2F&:tabs=no&:toolbar=yes&:animate_transition=yes&:display_static_image=yes&:display_spinner=yes&:display_overlay=yes&:display_count=yes&:loadOrderID=0 [cited August 8, 2015].Google Scholar As research and development becomes ever more reliant on industry-funded scientific investigations, it is important to understand the impact that this might have on the type of research being undertaken and, more importantly, how it is reported. In a recent analysis of 169 randomized controlled trials published between 2008 and 2009, 69 (41%) were industry-funded.15Lundh A. Krogsbøll L.T. Gøtzsche P.C. Sponsors' participation in conduct and reporting of industry trials: a descriptive study.Trials. 2012; 13: 146Crossref PubMed Scopus (40) Google Scholar Data analysis in 58% of the industry-sponsored trials (40/69) was provided either by the sponsor or an affiliated research organization without involvement of the academic authors. In 64 of these trials (93%), the manuscript was either written by or was approved by the sponsor.15Lundh A. Krogsbøll L.T. Gøtzsche P.C. Sponsors' participation in conduct and reporting of industry trials: a descriptive study.Trials. 2012; 13: 146Crossref PubMed Scopus (40) Google Scholar This is important as research sponsored or supported by industry is more likely to publish findings that support the use of the product.16Kairinos N. Pillay K. Solomons M. Hudson D.A. Kahn D. The influence manufacturers have on negative-pressure wound therapy research.Plast Reconstr Surg. 2014; 133: 1178-1183Crossref PubMed Scopus (22) Google Scholar, 17Lundh A. Sismondo S. Lexchin J. Busuioc O.A. Bero L. Industry sponsorship and research outcome.Cochrane Database Syst Rev. 2012; 12MR000033Crossref PubMed Scopus (803) Google Scholar Although bias in the academic literature is often attributed to “inducement” i.e. physicians might be compelled to use or endorse a device or drug because of a benefit supplied to the physician by the company,18Smieliauskas F. Conflicts of interest in medical technology markets: evidence from orthopedic surgery: conflicts of interest in medical technology markets.Health Econ. 2015 Apr; https://doi.org/10.1002/hec.3177Crossref PubMed Scopus (12) Google Scholar an alternate form of bias manifests itself in the form of selective publication and sponsorship of non-evidence-based discussion. If the scientific method is based on a need for critical appraisal and examination of negative results, this second form of bias is potentially an even more severe threat. It is becoming apparent that we, as academic surgeons, must become much more “savvy” in critically appraising the literature to be sure that the true effectiveness of any devices we use are reported and we need to listen more carefully when the voice of the reporter might have an investment in its outcome. Physicians play an important role in the device innovation process, both in their contribution to design and testing, and in the ability to participate in the critical appraisal of new concepts based on a life-long, in-depth study of the disease process being treated. This freedom to discuss the downfalls of any device and to critically debate its use is fundamental to the advancement of science. When that discussion is silenced and the scientific method subverted, patient safety is put at risk. The modern vascular community has witnessed the introduction of the Multilayer Stent into clinical practice and observed its evolution through being a potential “game changing technology” to its current status. It was not until Lowe's study, however, that the scientific method for evaluating the performance of this device was truly realized. In the Manchester series, it was appropriate to stop the pilot study when it became clear that there might be potential harm to the patient, and to report results to inform clinical practice. It is to be hoped that academic debate on the merits and/or pitfalls of the Multilayer Stent can now start in earnest. Tara M. Mastracci declares affiliation with Cook Medical Inc., for which she serves as a consultant and speaker, and Maquet Getinge Group, for which she serves as consultant and speaker. Since beginning work in London, UK, proceeds from these activities have been donated to the Royal Free Aortic Charity, which is used to fund educational and clinical activities for the Aortic Team at the Royal Free London. Dr Mastracci has no clinical experience with the Cardiatis Multilayer Flow Modulator.