The development, inter-rater agreement, and reliability of The CONSIDER Intervention Fidelity Checklist for clinical trials of complex interventions in healthcare.

Background: The Complex Interventions Design, Delivery and Receipt (CONSIDER) checklist was developed to facilitate systematic and reproducible identification of intervention fidelity in complex interventions controlled trials. This paper describes the development of the checklist and its evaluation in a cross-sectional reliability study with a purposeful sample of clinicians, researchers, trainees, and academics in complex interventions. The primary aims of this paper were to determine the internal consistency, inter-rater reliability, and agreement of the CONSIDER checklist. Secondary aims included exploring participants’ experiences using the checklist to identify items needing further development or modifications needed to improve the checklist’s usability.Methods and results: This study investigated the CONSIDER checklist’s inter-rater agreement, internal consistency, and inter-rater reliability across 3 clinical trial papers and 552 item and survey ratings by 15 independent raters (participants). Overall, percentages of agreement for the checklist items were high, with raters often choosing the same response options when assessing fidelity in the selected trial publications. The checklist items also have good reliability and internal consistency (the degree to which items on the checklist are correlated with one another), with Cronbach’s of 0.82. A high Kappa coefficient (0.861) confirmed the checklist’s inter-rater reliability and the extent to which the items described in the checklist are likely correct representations of the variables being measured. The high kappa coefficient also indicates that individual items could be distinguished between papers when participants used the checklist. Conclusion: CONSIDER is a reliable, internally consistent checklist for assessing intervention fidelity in complex intervention clinical trial papers. CONSIDER is, to the best of our knowledge, the first checklist developed to assess fidelity in trial publications for physical complex interventions and one of only a handful of checklists developed for assessing fidelity in published papers. The CONSIDER checklist can make an important contribution to the interpretation of findings from complex interventions trials and systematic reviews and meta-analyses.