Beyond Neyman–Pearson: E-values enable hypothesis testing with a data-driven alpha

A standard practice in statistical hypothesis testing is to mention the P -value alongside the accept/reject decision. We show the advantages of mentioning an e-value instead. With P -values, it is not clear how to use an extreme observation (e.g. P ≪ α ) for getting better frequentist decisions. With e-values it is straightforward, since they provide Type-I risk control in a generalized Neyman–Pearson setting with the decision task (a general loss function) determined post hoc, after observation of the data—thereby providing a handle on “roving α ’s.” When Type-II risks are taken into consideration, the only admissible decision rules in the post hoc setting turn out to be e-value-based. Similarly, if the loss incurred when specifying a faulty confidence interval is not fixed in advance, standard confidence intervals and distributions may fail, whereas e-confidence sets and e-posteriors still provide valid risk guarantees. Sufficiently powerful e-values have by now been developed for a range of classical testing problems. We discuss the main challenges for wider development and deployment.