Reproducibility and the future of MRI research

Most would agree that transparency in science is an essential goal, but there are differing opinions on how open our science really should be. The questions surrounding open science and research transparency are particularly relevant for fields like magnetic resonance, in which much of the innovation over the past 50 years has been driven by intellectual property, including patents and copyrighted software, and there is a tight link between industry and academia. Although many innovations manage to find their way to clinical MRI scanners under the current system, translational research can get bogged down and important details can get lost in translation. This is partly because many innovations are documented on a medium that is outdated—paper (or its electronic equivalent, the PDF). While these media are easy to distribute, they can leave a lot to be desired in terms of documenting the often-circuitous path from the birth of an idea, its development from theory, its implementation and the generation of data, to its ultimate publication. If we printed the first page of every paper ever published and stacked them on top of each other, the stack would reach the height of Mt. Kilimanjaro.1 As the paper mountain continues to grow,2 the general readability of papers inevitably decreases due to the use of jargon and the specialization and subspecialization of journals.3 This adversely affects the accessibility and reproducibility of research findings. The very word "paper," referring to a research product, is a relic from the previous millennium. It undersells the potential of a scientific publication. In the 21st century, we have the opportunity to redefine what the scientific paper means. The movement has begun with supplementing traditional journal publications with data and code that make the research process more transparent and facilitate the reproducibility of research findings. This is particularly relevant today, when a replication crisis is affecting several scientific disciplines.4-6 Fortunately for our journal, the field of magnetic resonance is built on a solid foundation. We are also fortunate that our field offers relatively broad accessibility both to the theory and to the experimental equipment. Articles in Magnetic Resonance in Medicine (MRM) undergo a rigorous vetting process that avoids some of the most obvious replication pitfalls. However, most of the articles submitted to MRM are still only "papers." The quality of the peer review process is often inhibited by the difficulty of accessing the data and the code used to produce the results presented in the manuscript. It doesn't have to be this way. For highly technical or empirically driven data science work, authors may unintentionally omit subtle, but important details about their methods or experimental setup. This can lead to a mismatch between the described and actual methodologies, which causes the reviewers to provide an inaccurate assessment of the work's value, in either a positive or a negative sense. The likelihood of such mismatch is highly reduced when the methodology is made openly available. By allowing the entirety of a work to be considered, research articles will become stronger and the review process will become more thorough, efficient, and objective. This will ultimately benefit all authors and readers. Although the explicit movement toward reproducibility, openness, and public provision of computational tools within our own community is relatively recent, the importance of these practices within the computational sciences has been an active topic of academic discussion for over 25 years.7-11 In particular, the reconstruction subspecialty of our field falls under this umbrella, and that community has led some of our initial efforts. The widespread use of many now-commonplace MRI technical methods, such as the nonuniform fast Fourier transform12 and compressed sensing (i.e., compressive sensing),13 can in part be attributed to their authors' foresight into the long-term benefits of promoting reproducible research by making their methods publicly available. Our community's fledgling initiatives in the area of reproducible research have been generally well-received. The recent survey on reproducibility conducted by the ISMRM Reproducible Research Study Group indicates that there are a significant number of researchers concerned about reproducibility and working toward promoting reproducible research.14 At the 2018 Editor's Forum organized by editors of the Radiological Society of North America, MRM was recognized as a leader among a sample of radiology and medical physics journals in the field of data and software sharing. The ISMRM community has recently revamped the MRHub website15 as a place where researchers can share code that drives or supplements existing publications. The MRM decision letters requesting revisions encourage authors to contribute to MRHub and other similar resources. We hope that these developments can continue to provide innovative supporting material that does not necessarily fit into the current publishing models. One example of an innovative research initiative is the challenge posed by the ISMRM Reproducible Research Study Group to reproduce a landmark MRI paper. The first choice was one of the seminal papers on SENSE published in MRM,16, 17 and we were surprised and encouraged by the number and the quality of the submissions.18 Scientists used different scripting languages (e.g., MATLAB, Python) and numerical libraries (e.g., PyNUFFT, BART, SigPy) to reproduce the original paper, and even some of the original authors welcomed the opportunity and reproduced their own work.19 Now that MRM researchers have started to reproduce each other's work in a public forum, the natural question arises: What venue(s) can best host such research objects? One possibility is a Jupyter Notebook,20 an interactive computing environment that can integrate data, text, code, and figures. While Jupyter notebooks were primarily designed for Julia, Python and R, they currently support more than 100 programming languages, including support for MATLAB/Octave, thanks to the community efforts. It is even possible to combine programming languages in the same notebook through the Script of Scripts environment.21 Additionally, projects such as MyBinder22 and NeuroLibre23 enable hosting notebooks that can be freely modified and re-executed through the web, offering a fully reproducible, "libre" path from data to figures. None of these initiatives would have been possible without our vibrant open-source community that is working together to improve the way we write, document, and share code. Hackathons (i.e., short, intensive, in-person meetings where developers collaborate on software projects) are a particularly effective way of bringing open-source developers together. The first hackathon for MR professionals24 took place in May 2019, immediately before the ISMRM Annual Meeting in Montréal. There, more than 50 researchers worked on 9 open-source projects, including MR visualization, quality control, data management and process automation, documentation, quantitative MRI tools, as well as the revamp of MRHub.25 To nurture and sustain this culture of openness, it is essential to ensure that researchers get appropriate academic credit when sharing their work. This will challenge academic promotion committees to go beyond the familiar legacy metrics (e.g., number of papers and citations, journal impact factor) to give credit for broader impact, such as how software packages or data sets are used. This is where outreach, communication, and community building can play an essential role in the open-science movement. The MRM communications channels (the Highlights blog and our social media channels) can provide a great service by spreading the message and creating a critical mass of open science advocates working on reproducibility. There is a common viewpoint that open science is necessarily incompatible with the protection of intellectual property. However, even the etymology of the word "patent" implies openness, and historically patents have offered an important alternative to companies hoarding trade secrets. However, patent disclosures can often delay publications, depending on the inventor's specific national laws. Although it is often true that patents and proprietary software can impose limitations on how (or at least when) the innovation is shared, even a limited degree of sharing has to be preferable to companies taking the trade secret route. When it comes to sharing data, there are also objective and regulatory limitations as a result of subject privacy concerns. Therefore, this journal has not imposed mandates on authors, and instead strongly encourages researchers to make available all aspects of their research that they realistically can. As this has become part of our culture, the degree of sharing code and data sets has become an increasingly important factor in reviewers' scores for manuscripts submitted to our journal. Metadata and aggregate statistics derived from data sets often go a long way when the raw data are not available. A release of limited-functionality, publicly available software in conjunction with a publication could improve reproducibility, while raising awareness for the commercial software. Many open-science advocates believe that as long as most scientific research is funded by taxpayers, all data, code, and results should be in the public domain. Skeptics worry that if we remove the possibility of patenting discoveries made with public money, it could have the unintended consequence of encouraging the best and brightest to leave academia for more lucrative jobs in industry, where important discoveries are more likely to remain as trade secrets or their public disclosures would be delayed further. The tug of war between open-science advocates and skeptics will continue for the foreseeable future, as scientists, funding agencies, and publishers try to figure out a way forward. Even within a single nation, patent law and rules imposed by funding agencies are very complex. For a truly international journal like MRM, it will be difficult to find a one-size-fits-all answer. However, few would disagree that reproducibility accelerates scientific progress, as well as its translation to clinical practice. As tectonic technological changes shape research in the 21st century, we need to be careful not to damage its foundations, and instead focus on aspects of open science where we can reach consensus. The open-science movement needs to pick its battles. The battle for reproducible research is one that it cannot afford to lose. The authors would like to thank Agâh Karakuzu for valuable discussions and feedback on this editorial.