Intergrader consistency reliability, achievement-referential validity, and expert-referential severity of adult student online peer grading

The study aimed to identify the reliability, validity, and severity of online peer grading of adult students, which remains uncertain from previous studies. There were 322 graduate students participating in the online peer grading activities for their presentations and discussions in the “Seminar I” and “Seminar II” courses. Both the peers and the teacher used a grading scale developed in the present study as a rubric to grade the students’ presentations. The peer grading and teacher grading scores can be inquired about and acquired from the peer grading system. Pearson’s correlation was used to examine the consistencies between peer and teacher graded marks (i.e., intergrader consistency reliability) and between peer graded and term project marks (achievement-referential validity). Intraclass Correlation Coefficient (ICC) was also used to measure the agreement between the two grading methods. A dependent-samples t-test was conducted to examine the differences between peer-graded and teacher-graded marks (expert-referential severity). The results revealed that online peer grading had sufficient intergrader consistency reliability and expert-referential validity, but insufficient expert-referential severity, when teacher was regarded as the standard-referential index. Online peer grading had sufficient achievement-referential validity when the term project score served as the achievement-referential index. Implications and recommendations are also presented in terms of academic theory and educational practice.