What question did this study set out to answer?

January 1, 2024Open Access

CausalBench: A Comprehensive Benchmark for Evaluating Causal Reasoning Capabilities of Large Language Models

Puntos clave

The aim is to assess the causal reasoning capabilities of large language models across various domains.
Introduced a benchmark covering textual, mathematical, and coding problems.
Developed problems examining causal understanding from four perspectives: cause-to-effect, effect-to-cause, cause-to-effect with intervention, and effect-to-cause with intervention.
Evaluated state-of-the-art large language models using the new benchmark.
Provided insights into current causal reasoning abilities of various large language models.
Demonstrated how causal reasoning performance relates to the generation of hallucinations.

Resumen

Causal reasoning, a core aspect of human cognition, is essential for advancing large language models (LLMs) towards artificial general intelligence (AGI) and reducing their propensity for generating hallucinations.However, existing datasets for evaluating causal reasoning in LLMs are limited by narrow domain coverage and a focus on cause-to-effect reasoning through textual problems, which does not comprehensively assess whether LLMs truly grasp causal relationships or merely guess correct answers.To address these shortcomings, we introduce a novel benchmark that spans textual, mathematical, and coding problem domains.Each problem is crafted to probe causal understanding from four perspectives: cause-to-effect, effect-to-cause, causeto-effect with intervention, and effect-to-cause with intervention.This multi-dimensional evaluation method ensures that LLMs must exhibit a genuine understanding of causal structures by correctly answering questions across all four dimensions, mitigating the possibility of correct responses by chance.Furthermore, our benchmark explores the relationship between an LLM's causal reasoning performance and its tendency to produce hallucinations.We present evaluations of stateof-the-art LLMs using our benchmark, providing valuable insights into their current causal reasoning capabilities across diverse domains.The dataset is publicly available for download at https://huggingface.co/ datasets/CCLV/CausalBench.

Me gusta

Guardar

Ver artículo completo