Inter-Rater Reliability Analysis in Performance-Based Assessment: A Comparison of Generalizability Coefficients and Rater Consistency

This study examines the reliability of a performance-based assessment designed to measure university students’ basic statistical skills within the framework of Generalizability Theory (GT). The participants were 80 students enrolled in a Guidance and Psychological Counseling program. They completed a two-hour exam consisting of 10 applied tasks, which were scored independently by two raters using a detailed analytic rubric. Data were analyzed using a fully crossed design (person × item × rater). The fully crossed design is characterized by every person providing responses to all items, with every rater scoring all student responses. Variance components were estimated via the maximum likelihood method, and 95% confidence intervals were obtained using a bootstrap procedure. Results indicated that 50.2% of the total variance was attributable to students, 25.6% to items, and 16.6% to raters. The relative generalizability coefficient was calculated as .98, while the absolute decision coefficient (Φ) was .81. Increasing the number of items and raters improved the Φ coefficient and reduced error variance. Findings demonstrate that true performance differences among students were reliably captured, although rater effects could not be entirely eliminated. The study demonstrates that rubric use, investment in rater training, and the adoption of a multi-task, multi-rater approach are critically important in performance assessments.