Toolkits for implementing and evaluating digital health: A systematic review of rigor and reporting

OBJECTIVE: Toolkits are an important knowledge translation strategy for implementing digital health. We studied how toolkits for the implementation and evaluation of digital health were developed, tested, and reported. MATERIALS AND METHODS: We conducted a systematic review of toolkits that had been used, field tested or evaluated in practice, and published in the English language from 2009 to July 2019. We searched several electronic literature sources to identify both peer-reviewed and gray literature, and records were screened as per systematic review conventions. RESULTS: Thirteen toolkits were eventually identified, all of which were developed in North America, Europe, or Australia. All reported their intended purpose, as well as their development process. Eight of the 13 toolkits involved a literature review, 3 did not, and 2 were unclear. Twelve reported an underlying conceptual framework, theory, or model: 3 cited the normalization process theory and 3 others cited the World Health Organization and International Telecommunication Union eHealth Strategy. Seven toolkits were reportedly evaluated, but details were unavailable. Forty-three toolkits were excluded for lack of field-testing. DISCUSSION: Despite a plethora of published toolkits, few were tested, and even fewer were evaluated. Methodological rigor was of concern, as several did not include an underlying conceptual framework, literature review, or evaluation and refinement in real-world settings. Reporting was often inconsistent and unclear, and toolkits rarely reported being evaluated. CONCLUSION: Greater attention needs to be paid to rigor and reporting when developing, evaluating, and reporting toolkits for implementing and evaluating digital health so that they can effectively function as a knowledge translation strategy.