Reliability study demonstrates that clinicians achieve reliable qualitative coding of pediatric survey comments, highlighting a feasible strategy for quality improvement.
Key Points
To assess the feasibility and inter-rater reliability of training pediatric clinicians to systematically code narrative patient experience survey comments for quality improvement initiatives.
Partnered with an urban freestanding pediatric hospital routinely collecting Child HCAHPS narrative survey comments.
Assembled a six-person coding team consisting of two qualitative researchers, two pediatric inpatient physicians, and two pediatric inpatient nurses who underwent standardized codebook training.
Evaluated independent coding across domains including valence, staff type, care setting, actionability, and patient safety flags using pooled and pairwise inter-rater agreement (Kappa).
Qualitative researchers showed excellent inter-rater agreement (pooled Kappa = 0.88), while clinician coders demonstrated very good overall agreement (Kappa = 0.68) and nurses showed good agreement (Kappa = 0.60).
Agreement was consistently high across groups for comment valence and mentioned staff type, with binary actionability yielding higher concordance than three-level actionability categorization.
Nurses demonstrated the highest sensitivity in identifying potential patient safety issues within the narrative comments compared to other coder groups.