What type of study is this?

This is a Quantitative Study study.

September 30, 2025Open Access

Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Tasks

View Full Paper

Ask AI

Bookmark

Authors

This paper evaluates LLMs for automated reasoning tasks, revealing critical uncertainty issues in formal verification.

LLMs show promise in democratizing automated reasoning, but they face a challenge in formal verification due to their probabilistic nature.
Systematic evaluation of five LLMs highlights a domain-specific accuracy impact of autoformalization from +34.8% to -44.5% on different tasks.
A probabilistic context-free grammar framework models LLM outputs, yielding a refined taxonomy of uncertainty signals across tasks.
Selective verification via uncertainty signals can drastically reduce errors in LLM-driven formalization, making it a reliable engineering approach.

Ganguly et al. (2025) studied this question.

View Full Paper

Ask AI

Bookmark

September 30, 2025Open Access

View Full Paper

Ask AI

Bookmark

Authors

This paper evaluates LLMs for automated reasoning tasks, revealing critical uncertainty issues in formal verification.

LLMs show promise in democratizing automated reasoning, but they face a challenge in formal verification due to their probabilistic nature.
Systematic evaluation of five LLMs highlights a domain-specific accuracy impact of autoformalization from +34.8% to -44.5% on different tasks.
A probabilistic context-free grammar framework models LLM outputs, yielding a refined taxonomy of uncertainty signals across tasks.
Selective verification via uncertainty signals can drastically reduce errors in LLM-driven formalization, making it a reliable engineering approach.

Ganguly et al. (2025) studied this question.

View Full Paper

Ask AI

Bookmark