Correcting for Multiple Comparisons in Statistical Analysis of Animal Bioassay Data

Rusyn et al. (2020) raise some issues regarding our analysis of the glyphosate bioassay carcinogenicity data (Crump et al., 2020). They claim we “seek to erect new barriers to chemical evaluation, and propose a very high bar to classify a chemical as positive in a rodent bioassay.” However, our analysis method is not new, as it was proposed independently by 3 groups 30 years ago (Farrar and Crump, 1988; Heyse and Rom, 1988; Westfall and Young, 1989) as a solution to the multiple comparison problem caused by many tumor sites being subject to statistical analysis in a typical bioassay. Rather than “erecting new barriers,” the analysis method described in these papers provides a solution to this problem by providing statistical analyses with the correct false positive rate. The multiple comparison problem is particularly acute with glyphosate because more than ten long-term bioassays of glyphosate have been conducted (Crump et al., 2020). We illustrate the importance of correcting for multiple comparisons in the glyphosate data by pointing out that IARC’s (International Agency for Research on Cancer, 2015) summary of the glyphosate bioassay data focused on pointing out tumors in specific bioassays for which the response was statistically significant (p < .05). The most significant response in both IARC’s and our analysis was for hemangiosarcoma among male mice in one bioassay (p = .0013). However, assuming that this tumor was not identified a priori, for which identification there is no evidence, but became of interest only because of the highly significant response, our analysis found that the true significance of this “most significant response” was p = .26 rather than p = .0013. That is, assuming exposure to glyphosate had no effect on tumor production, so that the IARC-cited p-value reflected only the random assignment of animals to dose groups, the probability of getting a p-value equal to or smaller that the smallest observed value of .0013 simply by chance for some tumor in the 10 bioassays we studied was .26. As “evidence” that IARC did not focus on statistical significance of individual tumors in evaluating the potential carcinogenicity of glyphosate, Rusyn et al. point out that of seven tumors showing p < .01 in our analysis, only one was cited by IARC as evidence of carcinogenicity. However, Rusyn et al. apparently failed to realize that four of these tumors occurred in bioassays not addressed by IARC. The remaining two tumors were not mentioned in IARC’s evaluation. Rusyn et al. acknowledge that multiple comparisons in analyses of animal bioassay data can be a problem but do not provide evidence of any technical problems with our methodology. The primary criticism posed by Rusyn et al. is that our methodology does not incorporate historical control data. However, as the US EPA (2005) cancer guidelines state, “considerations [based on historical controls] should be weighed on their own merits and clearly distinguished from statistical concerns.” There is nothing that would prevent historical control information from being considered once the basic statistical analysis is complete. In fact, our article used historical control data to complement our statistical analysis. Rusyn et al. note that several organizations have published guidelines for analysis of bioassay data without recommending our method. However, both US FDA (Lin and Rahman, 1998) and the US EPA (2005) acknowledged the multiple comparison problem and recommended or discussed crude approximate methods for dealing with it. These ad hoc methods apply specifically to a single two-species-two-sex study and would be inappropriate for a combined analysis of the many glyphosate bioassays. By comparison, our method is not ad hoc and provides correct p-values when applied to any number of bioassays, including a single bioassay. We agree with the statement Rusyn et al. attributed to Wasserstein et al. (2019) that p-values should not be used as a sole or even primary basis for decision-making. We also agree with Wasserstein that “a reasonable prerequisite for reporting any p-value is the ability to interpret it appropriately.” However, before a p-value can be interpreted appropriately, it must be correct, ie, it must reflect the correct false positive rate for the hypothesis under consideration. The statistical method used in our analysis of the extensive bioassay data on glyphosate provides p-values that satisfy this requirement by properly accounting for the multitude of tumors that require analysis by standard statistical methods. This letter was self-funded. K.C. and D.Z. served on the EPA Federal Insecticide, Fungicide, and Rodenticide Act Science Advisory Panel (SAP), which met to review an EPA document on glyphosate on December 13–16, 2016. J.H. testified before this panel on behalf of Monsanto and had a 1-year consulting agreement with Monsanto that extended from November 14, 2016 to November 13, 2017. However, this agreement was limited explicitly to his work related to the SAP. He has had no contact with, nor received any compensation, advice, or data from Monsanto related to this letter. The remaining authors have no conflicts of interest to report.