What question did this study set out to answer?

This research aims to establish a protocol for evaluating the quality of edge rankings in knowledge graphs based on varying levels of evidence.

March 17, 2026Open Access

Expert Evaluation Protocol for Evidence-Graded Knowledge Graph Ranking Quality

Read Full Paperexternally

Key Points

This research aims to establish a protocol for evaluating the quality of edge rankings in knowledge graphs based on varying levels of evidence.
Defined evaluation dimensions for edge ranking quality.
Utilized statistical analysis methods including Kendall's tau_b and Krippendorff's alpha.
Conducted pilot evaluations with 3 experts on 20 edges.
Achieved a Kendall's tau_b score of 0.58, indicating moderate agreement among experts.
Obtained a Krippendorff's alpha of 0.71, suggesting strong reliability in the evaluations.

Abstract

Paper 10 in the Prioris research series. Specifies a protocol for expert evaluation of edge ranking quality across evidence levels. Defines evaluation dimensions, statistical analysis methods (Kendall's taub, Krippendorff's alpha), and software specifications. Includes pilot results (tau=0. 58, alpha=0. 71) from 3 experts on 20 edges. Part of the Evidence-Graded Knowledge Graphs research programme (Paper 0: 10. 5281/zenodo. 19024611).

Connected Papers

Building similarity graph...

Analyzing shared references across papers

Discussion

Authors

Rogier Meulenaar

Actions

Institutions

Radboud University Nijmegen

References and Citations

Connected Papers

Building similarity graph...

Analyzing shared references across papers

Expert Evaluation Protocol for Evidence-Graded Knowledge Graph Ranking Quality

Key Points

Abstract

Citation Network

Connected Papers

Discussion

Authors

Actions

Institutions

References and Citations

Citation Network

Connected Papers

Discussion

Cite this study

Also consider

Also consider