Expert Evaluation Protocol for Evidence-Graded Knowledge Graph Ranking Quality

Key Points

• This research aims to establish a protocol for evaluating the quality of edge rankings in knowledge graphs based on varying levels of evidence.
• Defined evaluation dimensions for edge ranking quality.
• Utilized statistical analysis methods including Kendall's tau_b and Krippendorff's alpha.
• Conducted pilot evaluations with 3 experts on 20 edges.
• Achieved a Kendall's tau_b score of 0.58, indicating moderate agreement among experts.
• Obtained a Krippendorff's alpha of 0.71, suggesting strong reliability in the evaluations.