Answers are grounded in this paper and cite their sources.
We thank all the discussants for the careful reading and insightful comments. These discussions are very helpful to deepen our understanding of multiple testing beyond conventional error rate control and when it will be useful in practice. We are especially pleased to see the variety of perspectives and applications, suggesting that the debate on appropriate error rates is important and will hopefully continue. As we largely agree with the comments, this rejoinder can be viewed as an extension of the discussion, grouped by common themes raised by the discussants. In Maurer et al. (2023), we introduced the familywise expected loss (FWEL) and focused on additive and binary step loss functions. We pointed out that the FWEL specializes to the conventional familywise error rate (FWER) when using binary step loss functions. Furthermore, the FWEL specializes to the per comparison error rate (PCER) and the per family error rate (PFER) when using additive step loss functions for specific choices of the losses , as nicely illustrated by Benjamini et al. (2023). They argued against using the PCER and referred to the decision rule in the lower left panel of Figure 2 in Maurer et al. (2023) for potential consequences in terms of rejection thresholds becoming larger than α if the tests had to be performed separately. These consequences may be reasonable, however, as long as the implications of an erroneous rejection (i.e., the costs) are well understood. Technically, by reducing the loss values (e.g., compared to the off-diagonals for the FWER in Table 1) one would expect the rejection thresholds to increase beyond α. In the specific example of Figure 2, the prevalence is different between the two subgroups, leading to larger thresholds for testing an effect in the smaller subgroup (and making it easier to reject the associated null hypothesis) because the costs of incorrect decisions are smaller. Of course, if one feels uncomfortable with rejection thresholds being larger than α, then one can include additional constraints, as suggested by Benjamini et al. (2023) in reference to Heller et al. (2022). Losses for all possible decisions when both null hypotheses are true. (A): FWER. (B): 2-FWER. Losses for all possible decisions when both null hypotheses are true. (A): FWER. (B): 2-FWER. The three error rates above (FWER, PCER, PFER) were all introduced by Tukey (1953). We see it as an advantage that they can be embedded within a general class of error rates, as it helps clarifying to the practitioner the consequences of incorrect decisions. A summary of the FWER for null hypotheses, as shown in Table 1, shows more transparently the consequences of incorrect decisions than a statement like ‘reject at least one true null hypothesis.’ In Table 1, we have a similar summary of the k-FWER (for ) defined as the probability of rejecting at least k true null hypotheses (Lehmann & Romano, 2005). To see this and using the notation from our paper, let denote the index set of true null hypotheses for and k a positive integer. Setting the loss function and 0 otherwise then leads to the k-FWER. The FWEL also covers other error rates or can be used to tailor an error rate to a given problem at hand through a suitable choice of the loss function and decision rule. We particularly enjoyed learning from Brannath (2023) how the FWEL specializes to the populationwise error rate (PWER) recently introduced by Brannath et al. (2023) through a loss function that combines additive and binary loss functions in a novel way. For the disjoint subgroup analysis application discussed in our paper, the PWER reduces to the FWEL with an additive loss function and losses being proportional to the prevalence of subgroup i. The result from Brannath (2023) also shows how weights arise “naturally” in subgroup analyses through the prevalences . It is remarkable that the works by Brannath et al. (2023) and Maurer et al. (2023) in extending conventional error rates were triggered by similar applications. Agreeing on the costs (and gains) in other settings than subgroup analyses may be more challenging, although we believe that the FWEL remains applicable. Consider, for example, testing two unequally important variables, such as one primary and one secondary endpoint in a clinical trial. The FWER then regards both endpoint hypotheses as equally important and does not account for the impact on patients: an incorrect market approval of a new drug based on a false positive primary endpoint outcome is considerably worse (and leads to much higher losses) than an incorrect label claim based on a false positive secondary endpoint outcome. Similar observations were raised by LaVange et al. (2023) and previously by Jaki and Parry (2016). Existing multiple test procedures mask this problem when testing, for example, hierarchically ordered endpoints in a fixed sequence (Maurer et al., 1995), but arguably the FWER might not be the appropriate error rate to begin with. A related example occurs when conditional approval for marketing a new drug can be given before a decision on full approval is made. Conditional approval is granted based on a positive result in a surrogate endpoint (e.g., progression-free survival), whereas full approval is granted based on a primary clinical endpoint (e.g., overall survival). Obviously, the loss induced by an incorrect full approval is larger than that of an incorrect conditional approval alone and the loss of both together is larger than each single one (Hsu et al., 2023). As pointed out by Benjamini et al. (2023), weights have been introduced into the testing of multiple hypotheses a long while ago. There are particularly many applications of weighted tests to multiple endpoint analyses in clinical trials, where weights are typically selected for clinical reasons and not as the result of numerical optimization, all under the FWER framework (Westfall & Krishen, 2001; Bretz et al., 2009; Burman et al., 2009; Dmitrienko & Tamhane, 2013; Alosh et al., 2014). Controlling the FWER for drug applications has the advantage that it is a simple and universal approach to ensure that the probability of market approval of ineffective drugs or of incorrect or misleading claims remains small, but it does not account for the relative risks that come with false decisions. It will be interesting to see, whether and how this collective body of experience can be used to inform the losses in a formal decision framework underlying the FWEL when testing multiple endpoints. Brannath (2023) and LaVange et al. (2023) provided an excellent discussion of other clinical trial settings, where the need for multiplicity adjustment may depend on the specific application. The arguments can be subtle and in the following we focus on just one specific application. Consider the situation of a global clinical trial conducted across multiple regions that may serve as basis for product approval by different regulatory agencies. By doing so, it becomes unnecessary to conduct separate trials in different regions. However, it may occur that one regulatory agency requires one primary endpoint as the basis for product approval, while another agency requires a different primary endpoint, thus resulting in two different primary endpoint hypotheses . It is generally agreed that in such settings no adjustment for multiplicity across the two primary endpoints is necessary (ICH, 2017), although embedding it into the FWEL framework may offer additional insights as different viewpoints on the loss function can be taken. One possibility is to argue that each regulatory agency has its own loss function. For example, agency i could set and , . This would lead to level-α tests for by agency i and thus not control the FWER at the trial level. Noteworthy, would automatically be rejected by agency i, which is justified when assuming that approval for region i depends only on the primary endpoint hypothesis . In practice, may still contribute to regulatory decision making, for example, through drug labeling. If so, then a false rejection of would lead to additional losses, and a loss function other than the one above should be used. An entire different possibility is to argue that there is a single universal additive loss function (say, from a patient's perspective) with . Using level-α tests in each of the two regions essentially means using a conservative testing approach. Instead, thresholds like those shown in Table 2 of Maurer et al. (2023) can be used. In practice, however, one would have to consider similarities to off-label use and the motivation for using sub-additive loss functions. The situation is further complicated by the fact that regulatory agencies make their decisions based on the same data, which is different to the setting considered in our paper, since we assumed disjoint subgroups with independent observations. LaVange et al. (2023) and Banerjee (2023) advocated the use of Bayesian methods as an alternative approach to classical decision theory. LaVange et al. (2023) discussed a simulation-based framework for Bayesian designs relying on recent developments by Alt et al. (2022) for FWER control. Banerjee (2023) focused mainly on Bayesian modeling for complex spatial data and followed the approach by Müller et al. (2004) for FDR control. It is worth pointing out a result from Berger (1985, p. 159), which shows that minimizing the Bayes risk with a prior distribution and minimizing the posterior expected loss with a posterior distribution are equivalent problems. In other words, using either the frequentist decision criterion or Bayesian criterion leads to the same optimal rule. This result provides a Bayesian interpretation of our work by letting in equation (13) be a prior distribution and recognizing that the Bayes gain in that equation is derived using the same frequentist decision criterion as the Bayes risk in Berger (1985). Despite this link, one essential difference remains regarding the incorporation of weights into the error rate control. Both Alt et al. (2022) and Banerjee (2023) used decision rules based on thresholding relevant posterior probabilities. To be more precise, Alt et al. (2022) compared the posterior probability of at least one incorrect rejection under the global null hypothesis of no overall treatment effect with an adjusted threshold ϕ0 which is derived from the asymptotic distribution of this posterior probability in order to achieve FWER control. Banerjee (2023) compared the posterior probabilities of adjacent regions exhibiting different effects to a common threshold, adjusted to achieve FDR control. In both approaches, the individual hypotheses are treated equally, without any assigned weights. In contrast, with the proposed decision-theoretic framework we provide an explicit specification of loss functions via a finite actions formulation of a given multiple test problem, where weights are accounted for by a suitable choice of the loss values associated with incorrect rejections of individual null hypotheses. Different choices of loss functions lead to different error rates beyond the conventional FWER. In addition, the proposed decision-theoretic approach disentangles the incorporation of historical information (if any) via the prior and the incorporation of weights via the losses . We believe this disentanglement to be more transparent than specifying a prior that accommodates both the historical information and weights. The simulation-based approach suggested in LaVange et al. (2023) relies on sampling from a validation prior, which is a distribution over the parameter space under the global null hypothesis (i.e., the intersection of the null space of all individual null hypotheses). By doing so, the FWER is controlled only in a weak sense. This is in contrast to the conventional requirement of strong error rate control (i.e., under any constellation of null and alternative hypotheses) and which might have contributed to the result referred to in LaVange et al. (2023) that the approach by Alt et al. (2022) is more powerful than Holm's method. A naive extension could be to sample from validation priors under any constellation of null and alternative hypotheses which, however, may quickly become computationally cumbersome as the number of hypotheses increases. It would be of practical interest to extend this approach to take into account strong error rate control and unequal weights while providing decisions for individual hypotheses. We acknowledge that using a Bayesian approach could be attractive in certain applications. For example, in the spatial difference boundary detection problem discussed by Banerjee (2023), inference on spatial random effects is easier when using a Bayesian model than a frequentist model, as the author pointed out. Another example is the multiple decision problem introduced by Banerjee (2023) when investigating the set of hypotheses concerning all possible equal and unequal relationships among the effects. A Bayesian solution based on posterior probabilities of the individual hypotheses, as suggested by the author, seems rather intuitive. Compared to a set of hypotheses concerning one treatment effect at a time, however, it addresses a different experimental question. In the confirmatory subgroup analyses that motivated our work, hypotheses consisted of only one treatment effect at a time. Consider the simple case of two subgroups and let ϕ1 and ϕ2 denote the associated treatment effects. If we had used a Bayesian approach in our work, we would examine and to investigate the treatment effect in each subgroup separately. Instead, Banerjee (2023) would examine to investigate whether the treatment effect is the same between subgroups. The latter inference would not suffice in a confirmatory trial setting, as the objective then is to determine the drug effect in each subgroup. However, such inference might be well suited in exploratory settings with many subgroups, where the objective is to find any differentiable treatment effect. In conclusion, we are very grateful for all the comments on our paper, although space limitations restricted us to focus on a few common themes. Wang et al. (2015) suggested that “future research and discussions focus on the consideration of error rates more insightful than FWER; error rates that acknowledge two incorrect rejections are worse than one incorrect rejection, and some incorrect rejections are more serious than other incorrect rejections.” We are glad to see this topic being picked up (Hsu et al., 2023) and we believe that these discussions will stimulate new research beyond conventional error rate control.
No takes yet. Share an insight, caveat, or question.
Maurer et al. (2023) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: