Transparent reporting for reproducible science

The inability to reproduce key scientific results is a growing concern among scientists, funding agencies, academic journals and the public (Begley and Ellis 2012; Prinz et al. 2011). Studies examining research practices in neuroscience and other fields have highlighted problems with study design (Button et al. 2013), inadequate reporting of methods and results (Baker et al. 2014), and errors in statistical analyses (Lazic 2010; Strasak et al. 2007) and data visualization (Weissgerber et al. 2015). Each of these factors may contribute to irreproducibility. These observations have sparked discussion about the respective roles and responsibilities of authors, peer reviewers and journals in improving the quality of scientific literature (Nature Editorial, 2013; Nature Medicine Editorial, 2016; Collins and Tabak 2014; Landis et al. 2012). In principle, one can distinguish between efforts to improve the quality of the research itself and initiatives aimed at improving the quality of scientific reporting. In practice, however, these two factors are often intertwined. Initiatives and guidelines designed to improve the quality of research often contain recommendations for investigators at all stages of the research process, including preparing manuscripts for publication. Without following these recommendations, authors might inadvertently omit critical details when describing their methods and results, which could render even an excellent study to appear as low quality. The Journal of Neuroscience Research is introducing new policies, which are designed to ensure that papers contain essential information that reviewers and readers need to evaluate published studies. By having authors report complete, accurate and transparent scientific methods and statistical results; this policy will reduce inadequacies in experimental reporting and improve reproducibility and replication. These policy changes are in line with the National Institutes of Health Principles and Guidelines for Reporting Preclinical Research (NIHPG). This editorial provides an overview of these policies and highlights other resources that may help scientists to improve the quality of research and scientific data reporting. A reader's ability to critically evaluate original research depends on clear reporting of the study design and methods. Reporting should include a general overview of the study design, including the objective of the research, an explanation of methods that were used to reduce the risk of bias and a detailed description of the experimental protocol that will allow others to replicate the study. Many papers, unfortunately, are missing crucial information (Landis et al. 2012). The ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines, for example, were designed to improve the quality of reporting in animal studies (Kilkenny et al. 2010). Despite their widespread endorsement, an analysis performed two years after their release revealed that many journals had not fully implemented the guidelines (Baker et al. 2014). Articles published after the guidelines became available were more likely to report the species, sex and age of animals. However, only 20% of studies provided information on blinding. Fewer than 10% of articles reported whether the study was randomized or presented a power calculation. Similar challenges in implementing the CONSORT (Consolidated Standards of Reporting Trials) guidelines for human clinical trials have led investigators to propose that reporting templates may be more effective than checklists (Altman 2015). This approach may improve the quality of reporting in animal studies and other types of basic science research. Clear reporting of the study design is necessary, as it determines many aspects of the statistical analysis. Knowing whether the data are independent or non-independent is necessary to select statistical methods. Independent designs (Figure 1A) are analyzed using techniques such as unpaired t-tests or ANOVA, which assume independence. In contrast, repeated measures (Figure 1B) and clustered designs (Figure 1, Panels C-E) require analysis techniques that account for non-independence, such as paired t-tests or repeated measures ANOVA. Reporting of the study design is particularly important for animal and in vitro studies, as many experiments use different study designs for different outcome measures. Study design figures are valuable and underutilized tools for highlighting key features of the experimental design that affect the statistical methods. Study designs for independent and non-independent data. The figure illustrates study designs for independent (A), longitudinal or repeated measures (Panel B), Between group clustered (C), within group clustered (D) and between and within group clustered (E) designs. Inadequate reporting of statistical methods and misuse of statistical techniques are common in basic science research, even among articles published in top journals (Baker et al. 2014; Lazic 2010; Strasak et al. 2007). Problems with reporting may include, not presenting a power calculation (Strasak et al. 2007), failing to state which statistical test was used (Strasak et al. 2007), providing adequate detail about the test (i.e. paired vs. unpaired t-test) (Strasak et al. 2007), not stating whether the assumptions of the statistical tests were examined (Strasak et al. 2007; Weissgerber et al. 2015), or not stating how replicates were analyzed (Lazic 2010). Problems with statistical analysis include, using incorrect or suboptimal tests (Baker et al. 2014; Strasak et al. 2007), using mean and standard deviation or standard error to summarize data that were analyzed by non-parametric tests (Weissgerber et al. 2015), reporting p-values that do not match the test statistic (Garcia-Berthou and Alcaraz 2004; Nuijten et al. 2015), p-hacking (Head et al. 2015), and analyzing non-independent data as though they are independent (Lazic 2010). Along with addressing these problems, the scientific community is increasingly recognizing the need for better statistical education for basic scientists (Weissgerber et al. 2016a). This includes specific strategies for analysis and interpretation of small datasets, clustered data, and handling outliers, as well as learning to identify and avoid errors in data visualization and analysis that are common in the basic biomedical sciences. The methods used to determine sample size should be addressed in the statistical methods section. A recent paper highlighted low statistical power and small sample sizes as major obstacles to reproducibility in the neurosciences (Button et al. 2013). While some types of experiments depend on statistical analyses and power calculations, others do not (Vaux 2012). Power calculations traditionally have focused on reducing the likelihood of false negatives (Halsey et al. 2015). This can lead to the misleading impression that an underpowered study is only problematic if no effect is found. This perspective is incomplete, as it ignores the potential for false positives. Additionally, when the sample size is too small, samples that yield significant differences are usually extreme in some way and will overestimate the magnitude of the difference between groups (Halsey et al. 2015). The numbers of participants, animals or samples included in the original study should be clearly specified, with detailed explanations of the reasons for any attrition in the study. This includes providing information about how outliers were identified and handled in the analysis. A recent meta-analysis highlighted the problems with reporting of attrition in preclinical animal studies of stroke and cancer (Holman et al. 2016). In 64.2% of stroke studies and 72.9% of cancer studies, it was not possible to determine whether animals were excluded or did not complete the experiment. Among studies with clear evidence of attrition, most authors did not explain the reasons for attrition. This information should be included to demonstrate that the results are not affected by systematic biases that may be introduced by the exclusion or loss of samples. Simulation studies indicate that biased exclusion of a few animals can inflate the estimated treatment effect (Holman et al. 2016). The statistical methods section should also specify how non-independent replicates are handled in the analysis. Clustered data are frequently obtained in neuroscience research, but are often inappropriately analyzed or ignored (Lazic 2010). A cluster contains multiple observations, giving data a hierarchical structure (Galbraith et al., 2010). When analyzing these data, researchers often make the mistake of treating the data as if the observations were independent (i.e., pooling individual synapses to create a large dataset), which could increase statistical error. "Data reduction" is one of the most common strategies for analyzing clustered data (Galbraith et al. 2010). Replicates from a single subject or experiment (e.g., multiple synapses from the same slide, brain section, or animal) are converted into a single summary statistic (i.e. mean, median, etc.) and then analyzed using techniques that assume independence. While data reduction may be the only option for very small datasets, this approach reduces statistical power, as information is lost when the replicates are averaged. More sophisticated techniques, such as bootstrapping, permutation tests, and mixed models, may be valuable for investigators with larger datasets (Galbraith et al. 2010). Sufficient details about statistical tests should be presented when reporting study results. For studies using traditional (frequentist) statistical analysis, this includes the test statistic, degrees of freedom, p-value, and whether the test was one-sided or two-sided. The description of the statistical methods should provide enough details to enable a reader with access to the data to verify the reported results (Bailar and Mosteller 1988). An analysis of papers published in psychology journals between 1985 and 2013 observed that approximately half of papers included at least one p-value that did not match the test statistic and degrees of freedom (Nuijten et al. 2015). Large errors that may have altered the conclusion of the test were found in 12.5% of papers (Nuijten et al. 2015). Detailed discussion of these important topics is beyond the scope of this editorial. Many of these considerations have been incorporated in the updated Statistical and Data Reporting Guidelines within the Author Guidelines of the Journal of Neuroscience Research. Figures are often used to illustrate the most important findings from a study. A well-designed figure should convey information about the study design, in addition to showing the data. Selecting the right type of figure is essential (Table 1). Authors should consider the study design, type of outcome measure (i.e. categorical, continuous, etc.), and sample size when designing figures. Recent reports indicate that the common practice of using bar graphs to show continuous data for small sample size studies is a problem in the scientific literature (Saxon 2015; Weissgerber et al. 2015). Bar graphs are designed for categorical data; when used to display continuous data, bar graphs omit key information about the data distribution. This is problematic, as many different datasets can lead to the same bar graph and the actual data may suggest different conclusions from the summary statistics (Figure 2). The new guidelines recommend using figures that show the data distribution, such as univariate scatterplots (also called Cleveland dot plots (Cleveland 1993)), boxplots, violin plots, or kernel density plots when presenting continuous data. While line graphs are typically used to present longitudinal or repeated measures data, these figures provide limited information about data distribution and no information about whether responses vary among subjects. Showing univariate scatterplots or boxplots of differences for selected pairs of time points or conditions where important changes occur may alleviate this problem. Alternatively, recent publications have highlighted the importance of developing new techniques for the presentation of scientific results, such as interactive graphics (Krumholz 2015; Weissgerber et al. 2016b). We recently designed a free, web-based tool for creating interactive line graphs for scientific publications (Weissgerber et al. 2016b). This proof-of-concept tool shows how interactive alternatives to static graphics allow for additional exploration of published data. Tools such as this one have the potential to promote widespread use of interactive graphics and transform scientific publications from static papers into interactive datasets narrated by the authors. Small n: Univariate scatterplot Medium n: Box plot with data points overlaid Large n: Box plot, violin plot or kernel density plot Small datasets with 2 time points or conditions: spaghetti plot and univariate scatterplot showing change scoresa Larger datasets, or datasets with >2 time points or conditions: Interactive line graph or line graph and univariate scatterplot(s) showing change scores for time points where important changes occur Many different datasets can lead to the same bar graph. The full data may suggest different conclusions from the summary statistics. The means and standard errors for the four example datasets shown in Panels B-E are all within 0.5 units of the means and standard errors shown in the bar graph (A). P-values were calculated in R statistical software (version 3.0.3) using an unpaired t-test, an unpaired t-test with Welch's correction for unequal variances or a Wilcoxon rank sum test. In Panel B, the distribution in both groups appears symmetric. Although the data suggest a small difference between groups, there is substantial overlap between groups. In Panel C, the apparent difference between groups is driven by an outlier. Panel D suggests a possible bimodal distribution. Additional data are needed to confirm that the distribution is bimodal and to determine whether this effect is explained by a covariate. In Panel E, the smaller range of values for group 2 may simply be due to the fact that there are only three observations. Additional data for group 2 would be needed to determine whether the groups are actually different. Abbreviations: var, variance. Reprinted from Weissgerber et al. 2015 under a creative commons license. A recent article provides a valuable overview of graphic design principles that investigators should consider when creating figures for scientific publications (Duke et al. 2015). Numerous investigators have released free templates, tools or code that allows scientists to create graphics for continuous data. Resources for creating univariate scatterplots include, Excel templates (Weissgerber et al. 2015), instructions for GraphPad PRISM (Weissgerber et al. 2015) and code for creating these graphics in R (Ashander 2015; Marwick 2015) or SPSS (https://www.ctspedia.org/do/view/CTSpedia/TemplateTesting). R code is also available for investigators who want to create boxplots, with or without the data points overlaid (Ashander 2015). Authors who have clustered data can use the web-based tool provided by Pallmann and Hothorn to create customized boxplots with data points overlaid (Pallmann and Hothorn 2016) (https://lancs.shinyapps.io/ToxBox/). This application is easy to use and does not require any programming skills. Investigators can also use this tool to create box plots for studies with independent designs. A newly published article provides a detailed discussion about the advantages and disadvantages of different types of graphics that are frequently used to present categorical and continuous data in scientific publications (Rice and Lumley 2016). Code and links to resources for creating these graphics in Excel, R and Stata are available on the authors' website (http://faculty.washington.edu/kenrice/heartgraphs/). Other recent papers have provided detailed overviews of different strategies for visualizing differences in effect size (Wilcox 2006) and MATLAB scripts for visualizing fMRI data (Allen et al. 2012). A number of studies examining the research process have provided valuable insight into deficiencies in the study design, analysis and reporting of published research. Journal policy changes, such as the one being implemented by the Journal of Neuroscience Research, are essential components of a comprehensive strategy to promote transparent reporting and improve the quality and reproducibility of scientific studies. Initiatives designed to raise awareness about common problems in the literature, improve adherence to reporting guidelines and develop tools and templates to improve data visualization should all be a part of the solution. The authors have nothing to disclose. TLW drafted the editorial. All authors revised the manuscript for intellectually important content. TLW was supported by the Office of Research on Women's Health (BIRCWH K12HD065987). This work was supported by P-50 AG44170 (V.D. Garovic) from the National Institute on Aging.