Designing user-centered evaluations: Leveraging beta-testing results to develop process evaluation interview questions for the AMPLIFY program

Refining digital health programs with end-user feedback is essential throughout program lifecycles to ensure user-centered perspectives are incorporated during all program development. Results from beta-testing “think aloud” interviews exploring program engagement, satisfaction, motivation, and usefulness may be leveraged to support development of interview questions for a subsequent user-centered process evaluation implemented during program efficacy testing. We describe how results from beta-testing “think aloud” interviews were utilized to inform the design of user-centered process evaluation interview questions for implementation during program efficacy testing of AMPLIFY, a web-based diet and exercise program for older cancer survivors. Beta-testers (n = 15) were asked to complete three “think aloud” interviews via a videoconference platform over a 6-month period; 40 interviews were completed, accounting for early dropouts. Transcripts were analyzed using a multi-step process: 1) suggested refinements from beta-testing “think aloud” interview results were catalogued and organized according to whether the program feature was liked, disliked, or confusing to the beta-tester; and 2) suggestions for refinements were categorized based on feasibility to adopt (feasible/not feasible), the type of recommended change (formatting/content), and underlying improvement goal (engagement, satisfaction, motivation, usefulness). Refinements considered feasible to adopt were categorized into formatting refinements (changes to the presentation/organization of the program) and content refinements (additions, deletions, or changes to the information provided in the program). Refinements were also categorized by relationship to four process evaluation constructs: engagement, satisfaction, motivation, and usefulness. Results informed user-centered process evaluation interview questions for implementation during program efficacy testing; such that, subsequent semi-structured interview questions were developed based on these evaluation constructs and underlying improvement goals. The application of beta-testing results to refinements and process evaluation provide a potential guide for other researchers developing and evaluating digital health interventions.