Lessons from the influx of preprints during the early COVID-19 pandemic

Approximately 1 year since severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was first recognised, the COVID-19 pandemic has been characterised by widespread transmission, substantial strains on health infrastructure, and global losses of life numbering millions. However, the scientific response to the pandemic has taken place on an unprecedented stage of rapid, open-platform science, including considerable developments within preprint research literature. Uploading non-peer-reviewed articles as preprints can offer immediate accessibility without restrictions from potentially lengthy journal submission or publication processes. Such open-platform science has been specifically advocated during infectious disease outbreaks and other public health emergencies to guide timely and evidence-based public health responses.1Yozwiak NL Schaffner SF Sabeti PC Data sharing: make outbreak research open access.Nature. 2015; 518: 477-479Crossref PubMed Scopus (94) Google Scholar Although use of preprint repositories has become more commonplace within life sciences over the past 5 years,2ASAPbioBiology preprints over time.https://asapbio.org/preprint-info/biology-preprints-over-timeDate: July 25, 2020Date accessed: March 15, 2020Google Scholar the COVID-19 pandemic has resulted in large volumes of available preprint research, with over 6700 COVID-19-related preprints posted between January and April, 2020.3Fraser N Brierley L Dey G et al.Preprinting the COVID-19 pandemic.bioRxiv. 2020; (published online Sept 18.) (preprint).https://doi.org/10.1101/2020.05.22.111294Google Scholar When quantified against other globally notable outbreaks of emerging RNA viruses (eg, Ebola virus in 2014 and Zika virus in 2015; appendix p 1), to date, the speed of preprint responses to the COVID-19 pandemic appear enormously accelerated, increasing at rates well over 100 times higher (appendix p 2). Although an estimated average of 39·5 COVID-19 preprints have been posted per day during the pandemic, relevant preprints were posted only once every 10·5 days during the Ebola virus epidemic in 2014 (appendix p 4). Preprint responses to COVID-19 also occurred much earlier; the first preprint was posted to the bioRxiv repository 22 days after health authority notification of the initial cluster of cases in Wuhan, China. By contrast, no preprints addressing Ebola virus or Zika virus were posted until more than 6 months after first cluster notifications. Additionally, I compared COVID-19 preprints with those describing more recent infectious disease outbreaks of cholera (of which there has been an epidemic in Yemen since late 2016) and influenza (considering the onset of seasonal influenza in 2019 for comparability). Although research addressing cholera and influenza initiated sooner after the first reported clusters than did COVID-19 research (probably reflecting a level of constant research attention given the endemicity of these diseases), the availability of COVID-19 preprints accumulated at greater rates (appendix pp 2–4). Despite the fact that a larger increase in preprint accumulation is to be expected during this pandemic than in previous infectious disease outbreaks due to growth in preprint recognition,2ASAPbioBiology preprints over time.https://asapbio.org/preprint-info/biology-preprints-over-timeDate: July 25, 2020Date accessed: March 15, 2020Google Scholar the magnitude of acceleration for COVID-19 preprints is remarkable when considering the modest rates of preprint development over multiple seasonal influenza periods as a benchmark (appendix p 2). The value of preprints has been clear in the wide and timely distribution of important research findings that have advanced understanding of the biology of SARS-CoV-2, including phylogenetic and molecular characterisation of the virus and estimations of epidemiological parameters, such as the basic reproductive number. Evaluating the direct influence of preprints on active outbreak control is more difficult. Although some early and basic metrics of impact are available,3Fraser N Brierley L Dey G et al.Preprinting the COVID-19 pandemic.bioRxiv. 2020; (published online Sept 18.) (preprint).https://doi.org/10.1101/2020.05.22.111294Google Scholar assessing the true influence of preprints on the COVID-19 pandemic will require considerable bibliometric and meta-research efforts over the continuing months. The rise in preprint repositories has also narrowed the gap between academic and general audiences,4Fox F The preprint dilemma: good for science, bad for the public? A discussion paper for the scientific community.https://www.sciencemediacentre.org/the-preprint-dilemma-good-for-science-bad-for-the-public-a-discussion-paper-for-the-scientific-community/Date: July 17, 2018Date accessed: March 15, 2020Google Scholar providing public access to research material of immense topical interest. However, the non-peer-reviewed nature of preprints has proven to have limitations as well as benefits, allowing conclusions without scientific support to filter through various media channels in several notable cases. These preprints were highly criticised, with some now retracted as a result. One claim during the early COVID-19 pandemic was that the genome of the causative pathogen, SARS-CoV-2, had either naturally or artificially acquired genetic material from HIV on the basis of sequence similarity of observed inserts.5Pradhan P Pandey AK Mishra A et al.Uncanny similarity of unique inserts in the 2019-nCoV spike protein to HIV-1 gp120 and Gag.bioRxiv. 2020; (published online Jan 31.) (preprint).https://doi.org/10.1101/2020.01.30.927871Google Scholar However, this theory was quickly shown to be a simple false positive, resulting from the short sequence lengths in question.6Zhang C Zheng W Huang X Bell EW Zhou X Zhang Y Protein structure and sequence reanalysis of 2019-nCoV genome refutes snakes as its intermediate host and the unique similarity between its spike protein insertions and HIV-1.J Proteome Res. 2020; 19: 1351-1360Crossref PubMed Scopus (191) Google Scholar Another non-peer-reviewed, early access article from the Journal of Medical Virology suggested that the elapid snake species, the many-banded krait (Bungarus multicinctus), was a probable host of SARS-CoV-2.7Ji W Wang W Zhao X Zai J Li X Homologous recombination within the spike glycoprotein of the newly identified coronavirus may boost cross-species transmission from snake to human.J Med Virol. 2020; (published online Jan 22.)https://doi.org/10.1002/jmv.25682Google Scholar This theory was evidenced by similarities in codon usage between the virus and the krait, which were then shown to be artefactual through reanalyses, which emphasised that the codon usage of SARS-CoV-2 is almost identical to that of other mammalian betacoronaviruses.6Zhang C Zheng W Huang X Bell EW Zhou X Zhang Y Protein structure and sequence reanalysis of 2019-nCoV genome refutes snakes as its intermediate host and the unique similarity between its spike protein insertions and HIV-1.J Proteome Res. 2020; 19: 1351-1360Crossref PubMed Scopus (191) Google Scholar Although academic communities were quick to address these claims, their early availability as preprints meant that their findings were already widely disseminated within mainstream media articles and public perceptions.8Lee SM Shoddy coronavirus studies are going viral and stoking panic.https://www.buzzfeednews.com/article/stephaniemlee/shoddy-coronavirus-studies-are-going-viral-and-stoking-panicDate: Feb 4, 2020Date accessed: March 15, 2020Google Scholar Online searches for terms linking SARS-CoV-2 to HIV and snake hosts immediately increased following the respective preprint posting dates (appendix p 3). This situation further fuelled the uncontrolled and unfiltered spread of misinformation surrounding COVID-19, appropriately termed an infodemic.9Zarocostas J How to fight an infodemic.Lancet. 2020; 395: 676Summary Full Text Full Text PDF PubMed Scopus (1227) Google Scholar Even though it is tempting to think that public misinformation can only originate from non-scientific sources, misrepresented preprint research needs to be recognised as a potential driver of such infodemics. In response to these controversies, the bioRxiv repository added a disclaimer above all preprints emphasising that their contents were not peer-reviewed and should not be taken as definitive evidence in clinical decisions or in news reporting. Elsewhere, an official statement from leading figures denouncing misleading information about the origins of SARS-CoV-2 was made around 1 month into the pandemic,10Calisher C Carroll D Colwell R et al.Statement in support of the scientists, public health professionals, and medical professionals of China combatting COVID-19.Lancet. 2020; 395: e42-e43Summary Full Text Full Text PDF PubMed Scopus (156) Google Scholar and many academics have rightly continued to invest tireless efforts to combat conspiracy theory and reinforce public faith in science. However, further preventive ways of handling and filtering out unsound evidence from the large volumes of preprint material during future global emergencies are clearly necessary. This situation begs an important question; whose responsibility is it to regulate the use, transparency, and dissemination of preprints? Predictably, the answer is likely to be universal: that of authors to ensure that their preprint research is rigorous and presented objectively, that of preprint repositories to streamline opportunities for peer commentary (noting the dedicated efforts of prereview platforms such as Outbreak Science Rapid PREreview to encourage this), that of academics to provide such commentary in a timely and constructive manner, and that of the wider public readership to acknowledge the limitations of preprint research. In this case, a set of guidelines for good practice in understanding and interpreting non-peer-reviewed research for journalists and the general public would be a valuable resource to mitigate the spread of misinformation surrounding highly topical events. Example starting points for such guidance are provided in the panel. These guidelines broadly echo general considerations for interpreting the scientific method for non-academic audiences.PanelExample guidance for reading non-peer-reviewed preprint research for general audiences•Non-peer-reviewed research means that such work has not yet been reviewed by academic experts and, therefore, might not yet meet the required standards for scientific publication. Such work should be considered cautiously and not be treated as confirmed evidence.•Non-peer-reviewed research is not finalised. Specific statements, values, figures, and tables within such work are subject to change.•Non-peer-reviewed research can be prone to proofreading errors, including simple typographical errors and more substantial errors that might alter the interpretation of such work.•Scientific methods require collecting a consensus of knowledge from multiple studies and other sources of relevant evidence. If non-peer-reviewed research directly conflicts with another source, this does not mean that the findings are wrong. Further independent evidence is often needed for an informed scientific consensus to be made.•It is important to be especially mindful that such research is not yet reviewed when sharing or commenting through social media. Information from non-peer-reviewed research can be mistaken as confirmed evidence without proper context. Misunderstanding or misrepresentation of non-peer-reviewed research through social media can have harmful consequences (eg, influencing decisions and behaviours of members of the public away from guidance given by health authorities).•If unsure about exactly what conclusions can be drawn when looking to reference such research in a newspaper, magazine, blog, podcast, video, or other form of media, the authors represent the best initial point of contact. •Non-peer-reviewed research means that such work has not yet been reviewed by academic experts and, therefore, might not yet meet the required standards for scientific publication. Such work should be considered cautiously and not be treated as confirmed evidence.•Non-peer-reviewed research is not finalised. Specific statements, values, figures, and tables within such work are subject to change.•Non-peer-reviewed research can be prone to proofreading errors, including simple typographical errors and more substantial errors that might alter the interpretation of such work.•Scientific methods require collecting a consensus of knowledge from multiple studies and other sources of relevant evidence. If non-peer-reviewed research directly conflicts with another source, this does not mean that the findings are wrong. Further independent evidence is often needed for an informed scientific consensus to be made.•It is important to be especially mindful that such research is not yet reviewed when sharing or commenting through social media. Information from non-peer-reviewed research can be mistaken as confirmed evidence without proper context. Misunderstanding or misrepresentation of non-peer-reviewed research through social media can have harmful consequences (eg, influencing decisions and behaviours of members of the public away from guidance given by health authorities).•If unsure about exactly what conclusions can be drawn when looking to reference such research in a newspaper, magazine, blog, podcast, video, or other form of media, the authors represent the best initial point of contact. Growth in preprint repositories has been just one component of a shift towards open-platform research in outbreak responsiveness. Academic communities now have powerful channels to share findings with enough urgency to drive responses to public health emergencies, exemplified by the phenomenal boom in preprint research regarding the COVID-19 pandemic. However, preprint accountability and transparency remain important challenges. As the pandemic continues, repository-hosted preprints and other non-peer-reviewed research must be held under scrutiny, and academics should make sure that the limitations of preprints are made clear to non-academic audiences. Addressing these challenges will be a vital step as open-platform science further advances as an integral tool to face global crises. I declare no competing interests. I would like to thank Matthew Barnbrook, Jonny Coates, Anna Fowler, Alice Howarth, and Harry Parr for their comments and valued discussion during the development of this Comment. This Comment was supported by a MRC Skills Development Fellowship award (MR/T027355/1). The funders had no involvement in the writing of the manuscript or decision to submit for publication. Download .pdf (.31 MB) Help with pdf files Supplementary appendix