Assessing the Reliability of Grading Reflective Journal Writing

Journal writing has frequently been used in health care as an educational strategy to promote reflection. Although reflective journal writing is recognized as a valuable tool to assess students' learning, very little has been written about how to evaluate student journals. The purposes of this study were to establish criteria for evaluating the level of reflection in student journal writing and to examine the reliability of educators' grading of the journals. Fifty-eight students in a problem-based physical therapy program kept reflective journals documenting their perceptions of how they were adapting to problem-based education during an 8-week academic component of their third semester in a 24-month program. This study was conducted in three phases. In phase 1, 5 student journals were randomly selected from the 58 journals from which all identifying information had been removed. Using criteria developed by Boud et al to assess reflection, the journals were independently rated by three educators who assigned a separate grade for each half of the journals. Agreement among the raters as to the assignment of grades for each journal was established. The criteria for evaluating the reflective journals were then modified. In phase 2, each half of the remaining 53 student journals was independently rated by each educator In phase 3, a repeated-measures analysis of variance (ANOVA) was used to examine the interrater reliability of the educators' ratings of the journals. Split-half reliability was used to determine whether the students' reflective writing skills improved over the 8-week period. Interrater reliability as determined by the repeated-measures ANOVA was .68 (95% confidence interval=.49-.87), indicating acceptable agreement among the raters. The split-half reliability was .74, indicating a high correlation between scores on the first and second halves of the journals. These results suggest that the students did not improve in their reflection over the 8-week period. The results showed that the criteria that we established are reliable for evaluating reflection in journal writing.