Construction of a Korean Knowledge Conflict Dataset and Analysis of LLMs' Decision-Making Tendencies

Key Points

• This research aims to analyze the decision-making tendencies of large language models (LLMs) in knowledge conflict situations in a Korean language context.
• Developed a knowledge conflict evaluation dataset called KoConQA in a Korean setting.
• Conducted experiments on open-source LLMs using both zero-shot and few-shot learning environments.
• LLMs showed a stronger reliance on external context over inherent knowledge in both zero-shot and few-shot scenarios.
• Models were easily influenced by incorrect information present in the context.

Abstract