Inter-rater reliability of a scoring schema for assessing the treatment fidelity of preoperative physiotherapy education and breathing exercise training: a secondary analysis of LIPPSMAck-POP

Objectives To determine the inter-rater reliability of a scoring schema designed to assess the treatment fidelity of respiratory prehabilitation of education and breathing exercises taught face-to face by a physiotherapist, or via a booklet, prior to major upper abdominal surgery. Design Fully crossed (all assessors independently scoring all participants) inter-rater reliability analysis of a nested mixed-methods study within a randomised controlled trial. Setting Tertiary Australian hospital. Participants Twenty-nine adults scheduled for elective upper abdominal surgery. Interventions Participants were randomised to a single session of in-person preoperative education and breathing exercise training with a physiotherapist, or a booklet alone. On the fifth postoperative day participants were interviewed about their recall of their preoperative education via semi-scripted open-ended questions. Interviews were digitally recorded, deidentified, and transcribed. Transcripts were independently scored by four assessors using a standardised scoring schema. The inter-rater reliability of scoring a participant’s recall of breathing and coughing exercises taught either face-to-face or in a booklet alone was assessed with Krippendorff’s alpha. Results Across all 29 transcripts, the four assessors reached complete agreement for 22 participants (72%), with an alpha of 0.68 (95% CI 0.50–0.84). Agreement was complete for all 16 intervention-group transcripts (alpha 1.00), compared with 5 of 13 control-group transcripts (39%; alpha 0.36, 95% CI -0.01 – 0.62). Conclusion The scoring rule produced perfect agreement when applied to transcripts from participants who received face-to-face preoperative physiotherapy. These findings support the use of the schema as a reliable method for coding information recall in patients who receive preoperative physiotherapy education and breathing exercise training. Further work is required to establish its validity as a measure of intervention delivery, patient adherence, and clinical effectiveness. Trial registration : ACTRN12613000664741.