What does this research mean for the field?

While Large Language Models achieve similar or slightly better metacognitive accuracy and exhibit similar overconfidence compared to humans, they uniquely fail to adjust their confidence judgments based on past performance. Novelty: ClaimNovelty.NOVEL_FINDING. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

To evaluate the absolute and relative accuracy of confidence judgments produced by large language models compared to humans across domains of aleatory and epistemic uncertainty.

April 17, 2025Open Access

Quantifying Uncert-AI-nty: Testing the Accuracy of LLMs’ Confidence Judgments

View Full Paper

Ask AI

Bookmark

Authors

TCTrent N. CashDecision Sciences (United States)DODaniel M. OppenheimerDecision Sciences (United States)SCSara ChristieCarnegie Mellon University

Discussion

Member takes

Implication

Comparative evaluation reveals humanlike metacognitive accuracy alongside overconfidence in large language models, indicating severe limitations in adapting confidence to past performance.

Key Points

To evaluate the absolute and relative accuracy of confidence judgments produced by large language models compared to humans across domains of aleatory and epistemic uncertainty.
Assessed four large language models (ChatGPT, Bard/Gemini, Sonnet, and Haiku) alongside human participants across five experimental tasks.
Tested aleatory uncertainty using NFL outcome predictions (Study 1; n = 502) and Academy Award predictions (Study 2; n = 109).
Tested epistemic uncertainty through Pictionary gameplay (Study 3; n = 164), trivia questionnaires (Study 4; n = 110), and campus-specific university knowledge (Study 5; n = 110).
Large language models achieved absolute and relative metacognitive accuracy levels comparable to, and slightly exceeding, human participants across both aleatory and epistemic domains.
Models demonstrated systematic overconfidence, mirroring widespread human calibration biases.
Unlike human participants, models—most notably ChatGPT and Gemini—consistently failed to update or calibrate their confidence judgments in response to prior performance feedback.

Cite This Study

Cash et al. (2025) studied this question.

synapsesocial.com/papers/6a0da3decae7912d2fa524c9 https://doi.org/10.31234/osf.io/47df5_v3

View Full Paper

Ask AI

Bookmark

Implication

Comparative evaluation reveals humanlike metacognitive accuracy alongside overconfidence in large language models, indicating severe limitations in adapting confidence to past performance.

Quantifying Uncert-AI-nty: Testing the Accuracy of LLMs’ Confidence Judgments

Discussion

Member takes

Implication

Key Points

Cite This Study

Quantifying Uncert-AI-nty: Testing the Accuracy of LLMs’ Confidence Judgments

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider