Examining the Validity of the Implementation Quality Rubric for Simulation (IQR-SIM) for Assessing Implementation Quality of Simulation-based Programs.

INTRODUCTION: The lack of reliable tools for measuring implementation processes and outcomes weakens confidence in study findings and strategy effectiveness. This study aimed to gather validity evidence for the previously published Implementation Quality Rubric for Simulation (IQR-SIM) from 2 sources: response process and internal structure. METHODS: The IQR-SIM is structured into 3 phases of implementation, each divided into themes. Within these themes are items representing actionable steps for effective implementation. It employs a 3-point rating scale (1, 2, and 3), along with nonscored options "don't know/can't assess" and "not applicable." Semistructured interviews were conducted to collect detailed program descriptions from international simulation centers. Three trained raters evaluated these interview-derived implementation data using the IQR-SIM. The frequency of using the scores and the nonscored options was calculated to evaluate rater use of the scale. Inter-rater reliability was assessed to examine variability in ratings, while internal consistency reliability was analyzed to determine whether all items measured the same construct. RESULTS: Twenty simulation-based programs from 7 international simulation centers participated. Phase 1 (stakeholder engagement and context exploration) had the highest proportion of fully accomplished items (54.6% received a rating of 3/3). Theme 1b (Obtaining buy-in from relevant stakeholders) was predominantly rated as fully accomplished (68.3%). Item 9 (planned for human resource needs) received the highest proportion of rating 3/3 (83.3%). Internal consistency reliability was good (Cronbach alpha=0.85). Inter-rater reliability analysis showed a moderate intraclass correlation coefficient (ICC=0.57). ICC values between raters ranged from 0.39 to 0.52. CONCLUSIONS: Multisite validity evidence regarding the response process and internal structure of the IQR-SIM was established, supporting its use in evaluating the implementation quality of simulation-based programs. The study demonstrates that the IQR-SIM can be applied across diverse simulation centers and program types, enhancing its broad use for implementation evaluation.