Measuring guideline-concordant care when recommendations require clinical judgment: a mixed methods approach

Introduction: General practitioners (GPs) face challenges in diagnosing and managing work-related mental health conditions. In Australia, this prompted the development of a national clinical guideline (the Guideline) to support high-quality care. However, measuring the implementation of recommendations that require clinical judgment remains methodologically challenging. Objective: This study describes an approach to developing and testing a guideline-specific tool to assess concordance with recommendations in the context of an implementation trial. Methods: We used an exploratory sequential mixed methods design. Semi-structured interviews were conducted with 20 individuals with lived experience of work-related mental health conditions. The interview data informed personas and case studies, which were converted into 18 video vignettes simulating GP–patient consultations. Each vignette was mapped to a checklist of clinical indicators from the Guideline and paired with one multiple-choice and two short-answer questions. A scoring rubric (0–3 points per vignette) was developed and pilot-tested with three groups: GPs familiar with the Guideline, GPs unfamiliar with it, and medical/research students. Vignette content and questions were refined for internal consistency before use in a national implementation trial. Results: Internal consistency was acceptable (Cronbach's α = 0.74). In the IMPRovE trial, participants completed three vignettes per time point (score range: 0–9). Inter-rater reliability was almost perfect (Cohen's κ = 0.92) from 873 responses across 97 GPs. Conclusions: This study offers a rigorous, context-sensitive method for evaluating guideline-concordant care where recommendations require clinical judgment. Successfully implemented in a national trial, this approach offers a replicable method for evaluating guideline implementation where recommendations require nuanced clinical judgment. IMPRovE trial registration: ACTRN12620001163998, November 2020. Spanish abstract: http://links.lww.com/IJEBH/A597