Calibration Feedback with the Practical Scoring Rule Does Not Improve Calibration of Confidence

People are generally overconfident in their probabilistic judgements of future events or the state of their own knowledge. Some training methods have proven effective at reducing bias, but these often involve intensive training sessions with experienced facilitators. This is not conducive to a scalable and domain-general training program for improving calibration. In two experiments, we examined the effectiveness of a calibration training paradigm based on the Practical scoring rule, a modification of the logarithmic scoring rule designed to be more intuitive to facilitate learning. We examined two training regimes in comparison to a control group: performance feedback and outcome feedback. Participants were tasked with selecting which of two world urban agglomerations had a higher population and to provide their confidence level. The outcome feedback group received information about the correctness of their choice on a trial-by-trial basis as well as a summary of their percent correct after each experimental block. The performance feedback group received this information plus the Practical score on a trial-by-trial basis and information about their overall over- or under-confidence at the end of each block. We also examined whether Actively Open-Minded Thinking (AOMT) was predictive of calibration and its change across blocks. We found no improvement in calibration due to either training regime. Good calibration overall was predicted by AOMT, but not its change across blocks. The results shed light on the generalizability of other findings showing positive effects of performance training using the Practical scoring rule.