Too Much Left Unsaid? Mean Differences and Rational Decision-Making

We often want to choose the intervention that most effectively achieves our goals. But what are the most effective interventions? Here, we often turn to advice from scientists who report the effectiveness of tested interventions, using the mean difference: the difference in expected outcomes given a tested intervention and given a control intervention. I identify a problem with this practice: even perfectly accurate mean differences omit information about how interventions change the probability of an outcome of interest, and this omitted information can be vital for decision-makers to choose the best intervention—the one they would rationally choose if they were to learn all the information available in studies about how the interventions change the probability distribution over an outcome of interest. These interventions’ mean differences omit some of this information and, in doing so, may fail to inform the decision-maker sufficiently to identify the best intervention. That is the bad news. The good news is: I identify sufficient conditions for mean differences to inform rational agents sufficiently (specifically, those agents abiding by expected utility theory or risk-weighted expected utility theory). Based on my results, I advise how researchers and decision-makers can use mean differences to facilitate rational decision-making.