Comparison of Peer, Near-Peer, and AI-Assisted Methods to Faculty Grading of SOAP Notes

OBJECTIVE: The objective of this study was to determine which approach (peer, near-peer, Chat- Chat-Generative Pretrained Transformer [GPT] Plus, or an inhouse implementation of a GPT-4-based application referred to as GRADES) was closest to faculty grading of Subjective, Objective, Assessment, Plan (SOAP) notes. METHODS: Second-year student pharmacists (n = 83) completed a practice SOAP note. Five methods were used to grade the SOAP notes: faculty, peer, near-peer, ChatGPT Plus, and GRADES. Variability in rubric scores among grading methods was analyzed using Friedman 1-way repeated measures analysis of variance with post hoc testing. Time was tracked for each grading method to evaluate efficiency. RESULTS: The median scores and IQRs for each grading method were as follows: faculty (65% [56%-71%], peer (78% [62%-88%]), near-peer (77% [72%-86%]), ChatGPT Plus (87% [80%-91%]), and GRADES (66% [57%-73%]). Peer-, near-peer, and ChatGPT scores were statistically different from faculty scores. Scores from GRADES were not significantly different from faculty grading. CONCLUSION: The use of an inhouse implementation of GPT-4 (GRADES), with faculty oversight, resulted in rubric scores similar to those of a faculty grader.