What does this research mean for the field?

DaoEval provides the first structured evaluation framework for assessing large language models on Daoist knowledge, addressing the limitations of general-purpose benchmarks in evaluating scripture provenance, cross-sect interpretation, and ritual fidelity. Novelty: ClaimNovelty.METHODOLOGICAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

To create an open evaluation framework for Daoist-knowledge large language models (LLMs) addressing the gaps in existing evaluation suites.

May 19, 2026Open Access

DaoEval v0.1 — An Open Evaluation Framework for Taoist-Knowledge LLMs

Key Points

To create an open evaluation framework for Daoist-knowledge large language models (LLMs) addressing the gaps in existing evaluation suites.
Developed a JSONL schema with 500 samples for evaluations.
Implemented ABCD confidence grading and annotation protocols for three models.
Structured the framework around scripture provenance, cross-sect interpretation, and ritual fidelity.
DaoEval provides a comprehensive and structured evaluation approach for Daoist-knowledge LLMs.
Fills the gaps in existing evaluation methods by addressing specific needs in Chinese religious studies.
Facilitates better assessment of LLM applications in the humanities sector.

Abstract

DaoEval is the first open evaluation framework specifically targeted at Daoist-knowledge large language models. v0.1 includes: 10-chapter whitepaper (full text), 500-sample JSONL schema with sample examples, ABCD confidence grading, base-text and ID conventions, and a three-model cross-validation annotation protocol. Existing evaluation suites (MMLU, CMMLU, C-Eval) cover Daoism very thinly and only via multiple-choice questions. DaoEval is the first structured open framework covering scripture provenance, cross-sect interpretation, and ritual fidelity — three dimensions essential for any serious humanities application of LLMs in Chinese religious studies. This deposit serves as the first formal citable archive of the DaoEval framework, complementing the GitHub repository at lius-cc/lius-cookbook and the live whitepaper at https://lius.cc/llm/whitepaper. License: CC0 1.0 Universal (public domain dedication). Project home: https://lius.cc/llm Whitepaper: https://lius.cc/llm/whitepaper Cookbook: https://github.com/lius-cc/lius-cookbook

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper