Studies of Methods and Interventions for Judgment Accuracy: Methodological Guidelines

Many areas of society require humans or machines to make probabilistic judgments about the world, but the literature shows our probabilistic judgments are often fallible. The consequences can include fatal misdiagnoses, false death sentence convictions and other severe outcomes. To address this problem, researchers have studied various methods to assign probabilities (i.e. probabilistic methods) and other interventions to improve accuracy, such as placing participants in teams or under conditions of accountability. Yet not all study designs are equally good: some can be invalid or wastefully inefficient, for instance. This article outlines various designs for the study of methods and interventions to ensure or improve accuracy, and it provides some methodological guidelines. First, it outlines and evaluates some measures of accuracy, such as calibration and Brier score accuracy, since studies of accuracy need to appropriately measure it. Then, it delineates four types of study design: a priori, simulation-based, researcher-based and training-based. Some merits and limitations of each type are listed, and these sometimes entail that empirical accuracy tests are necessary. Guidelines qua desiderata are provided for these tests, such as appropriate use-case specification, sufficiently large sample sizes and best-practice questionnaire design, where relevant. Additionally, to facilitate the efficient and valid testing of methods and interventions, some code is described to run simulation-based power analyses of required sample sizes for questions and participants. Future studies may wish to utilize, critique or revise these guidelines for the sake of advancing the study of judgment accuracy.