Answers are grounded in this paper and cite their sources.
(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Post-hoc calibration is usually fitted and scored against one label per item, and where annotators disagree that label is a majority vote. A 2026 study argues that this is a structural error: temperature scaling fitted to votes settles on temperatures that are too low, looks calibrated against the votes, and stays overconfident against the distribution of annotator labels, with errors that grow with annotation entropy; the best calibrator it fits to the annotator distribution, Dirichlet-Soft, cuts true-label calibration error by 55 to 87 percent. In this analysis the author reviews and synthesises published results to show that the finding is correct and that its framing is one-sided. Calibration is a relation between a forecast and a named random variable, and under disagreement there are at least four candidates: the vote, the label of a randomly drawn annotator, the label of a member of a target population, and an outcome fixed by later evidence. Each result in this literature is calibrated against a different one. A model that reproduces the human distribution exactly is badly miscalibrated against the majority, by the same logic that makes vote-fitted temperature scaling miscalibrated against annotators. The annotator distribution is not a clean target either: re-collecting 100 labels per item changed the majority label on a quarter to a third of low-agreement NLI items; adjudication traced most disagreements in one retinal grading study to missed lesions and artifacts; and the two medical benchmarks in the 2026 study use synthetic annotators whose confusion depends only on the consensus class. Re-reading that study's tables, the author finds that when temperature scaling is refitted to annotator labels, ECE falls by 48 to 87 percent while the soft-target Brier score falls by 2 to 11 percent. The paper consolidates the measured values in Table 1 and Figure 1, states a target-declaration procedure as Algorithm 1, and specifies six tests that would say which target a deployment should be calibrated to and how much of the measured gap survives it.
No takes yet. Share an insight, caveat, or question.
Pranay M. Mahendrakar (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: