Cultivating critical thinking ability is an essential goal of higher education. Peer review is an effective means of critical thinking training widely used in open-ended assignment. However, there is a challenge in the teaching practice due to inexperience, subjectivity, and randomness of students’ grading, resulting in low accuracy. To address this issue, we introduce PAStudio, a peer assessment pedagogical tool. It utilizes a pairwise comparison method based on a binary system to significantly reduce the grading difficulty for inexperienced students. It requires students to evaluate projects across multiple dimensions using predefined grading criteria, thereby reducing the subjectivity and complexity of the grading process. To better assist students in iterative project improvement and manage the workload of reviews in PAStudio, we have designed an assessment workflow based on the Swiss system for multiple rounds of pairwise comparison. In this workflow, as reviewers, the student's assessment rounds are acceptable, and the assessment workload for each round is limited. As the submitter, the student receives two constructive comments from different peers. The process forms a formative feedback loop. In PAStudio, we have also developed functionality to avoid superficial comments using the Bidirectional Encoder Representations from Transformers (BERT) model. As the reviewer writes comments, the system rates them on a five-level scale. A comment rated ”Fail” cannot be submitted to the system. To calculate a project's score accurately, we propose an algorithm based on Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS). This algorithm considers the results of all rounds and the opponent's strength to calculate the assignment's overall performance comprehensively. PAStudio incorporated gamification elements such as dynamic leaderboards and badges to boost competition and engagement and enhance assignment quality. We analyzed learning data from the Swift Language Programming course from 2023 to 2024. When measured using Krippendorff's Alpha, the inter-rater reliability between instructor and peer grading shows a good level of agreement, demonstrating the reliability of the peer grading mechanism. Furthermore, peer grading is significantly associated with enhanced students’ assignment performance and learning engagement, as measured by Kruskal-Wallis test. The results of the course questionnaires indicate that most students perceived the peer assessment approach positively and have greatly benefited from this method.
Zhang et al. (Mon,) studied this question.