Steps for Planning Health Program Evaluations: From Program Logic to Data Collection and Reporting Plans

What is already known on this subject Evidence of effective health programs is very important, however relatively little is generated within rural contexts. Clinicians and health program managers are often well placed to lead evaluation activities but may not be familiar with evaluation methods. What this paper adds This paper provides an overview of a simple step-by-step process for planning health program evaluation. It demonstrates a method for developing data collection and reporting plans from a program logic document. There is a lack of evidence generated by health program evaluation in rural contexts [1, 2]. Evaluation of health programs is important for determining outcomes and impact [3], providing evidence for policy and funding decisions [4], and to drive continuous improvement of health programs that address inequalities in rural health outcomes [5, 6]. There are several challenges to healthcare evaluation in rural settings [1, 2, 4-6]. These can include resource limitations such as shortages of healthcare professionals, facilities, infrastructure, and funding, as well as geographical barriers such as vast distances and transportation issues, which may impede stakeholder engagement and data collection processes. Program evaluation must be tailored to the unique cultural and social contexts of rural communities. For example, qualitative data from interviews, video and audio recordings, and written feedback can provide useful evidence of outcomes and experiences of receiving care but may not be deemed appropriate in some communities. The views and support of Elders and other community leaders can guide the design and implementation of culturally safe and welcoming program evaluation activity. While a detailed explanation of these issues and management strategies is beyond the scope of this article, suggestions for approaching some of these issues are noted, and more information about managing these issues can be found. Given these challenges, rural clinicians and health program managers are often well placed to lead the evaluation of health programs. However, they may not be familiar with evaluation methods. This article provides an overview of a simple step-by-step process for planning health program evaluation and demonstrates a method for developing data collection and reporting plans from a program logic document. Develop a data collection plan. Two separate tables can be used to operationalise a Data Collection Plan from a Program Logic. Figure 1 provides a simple worked example of how these tables were used in planning an evaluation of the example Health Care Navigator program. The first table is a worksheet (see Figure 1, Step 3a). This worksheet displays the key elements identified from the Program Logic in the rows (e.g., Outputs, Outcomes short term, Outcomes medium term and long term). Step 3a shows the following information in columns. How information will be collected. In this example, demographic and clinical information will be collated from a client/patient record system, some specific measures such as Goal Attainment Scales, Camberwell Assessment of Need/Partnership Health (CAN/PIH) tools and Patient Reported Experience Measures (PREM) will be used. Clinical outcomes will include average Haemoglobin A1c (HbA1c) for people with diabetes and Girth for people who presented with cardiovascular disease (CVD)/overweight. The Kessler K5/K10 and EQ5D-5 L scales will be completed to measure mental wellbeing and quality of life. Interviews were planned with consenting clients/patients and carers. Where a small sample and limited statistical power prevents statistical analyses, case summaries or vignettes can present individuals' stories of the outcomes and experiences they achieved through the program via video or text/photos. Overall, methods must be feasible, acceptable and ethical for clients/patients and staff. Data must be stored securely and in a format that enables later analysis (e.g., password protected file storage, data in a spreadsheet). The second table is the final Data Collection Plan (Step 3b). This can be used to present the information generated in the first worksheet in a clean format that can be implemented by the evaluation team. The Data Collection Plan (3b) table lists each of the evaluation/outcome measures and methods as rows, while the columns identify Who will provide the information (e.g., clients/patients, caregivers), Who will collect the information (e.g., particular program staff member) by When the information will be collected (e.g., due by dates) and a brief explanation of What information is being collected (e.g., What the tool measures). From program logic to data collection plan—Brief example worksheets for a Health Care Navigator program. Consider additional presentation formats (e.g., Video vignettes, case vignettes/examples, photos, reflective journals, presentations…). Note. PHN = Primary Health Network, HCN = Health Care Navigator, HbA1c = haemoglobin A1C/glycated haemoglobin, Girth cm = Waist circumference centimeters, CAN/PIH = Camberwell Assessment of Need/Partnership Health, PREM = Patient reported experience measure, DE/GP = Diabetes Educator/General Practitioner, EP/GP = Exercise Physiologist/General Practitioner, OOS = Occasions of service, QOL = Quality of life. Report findings. Development of a clear reporting plan can be helpful in confirming the data to be collected, analyses required and in determining the content and structure of reports. This plan can be helpful before commencing data collection. Figure 1 (Step 6) shows an example Reporting plan for the Health Care Navigator program. The example Reporting Plan is presented in a table with the rows containing a Description of the program, and then Inputs, Activities, Outputs, and Outcomes—short, medium, and long term as rows (as per a program logic). The column can include information to present in the program evaluation report. In this example, the key information about the program, its development and implementation, and then achievements, including outputs and outcomes as listed in the Step 3b Data Collection Plan are noted. Evaluation of health programs is recognised as essential for providing evidence that can ultimately address inequality of health outcomes in rural communities [4-6]. However, this evidence is lacking for many rural health services [1, 2]. Our paper shares a simple introduction to operationalising a data collection and reporting plan from program logic. This method has been successfully implemented across several rural-based health and wellbeing programs, including a multidisciplinary paediatric outreach clinic, peer navigator, diabetes mentoring, Sense rugby (a rugby skill program for children with disability), as well as a school-based reading program. It may be useful to leaders in rural health who are considering program evaluation. Future research will evaluate and further develop this method with a focus on rural health programs. Matt Thomas: conceptualization, writing – original draft, methodology. Justine Summers: writing – review and editing. Sherryn Honeywood: writing – review and editing, methodology. Alyssa Fitzgerald: methodology, writing – review and editing. Donna Ambler: methodology, writing – review and editing. Kylie Falciani: methodology, writing – review and editing. Amanda Cook: methodology, writing – review and editing. Kelly Smith: methodology, writing – review and editing. Dean Bright: methodology, writing – review and editing. Michelle Lindsay: methodology, writing – review and editing. Catherine Sanford: methodology, writing – review and editing. We would like to acknowledge James Knight, Sally Hegvold and Raechel Nimmo (Communications Team, Marathon Health) for their assistance in formatting the content of this paper. We are grateful to the Managers, Clinical and Team Leads and staff across Marathon Health programs for their commitment and effort in implementing program evaluation and continuous improvement activities. Finally, we would like to express our gratitude for the engagement and support of the many individuals and organisations who have partnered with us to support program evaluation and the development of our capabilities in this area. Open access publishing facilitated by Charles Sturt University, as part of the Wiley - Charles Sturt University agreement via the Council of Australian University Librarians. The authors declare no conflicts of interest. The authors have nothing to report.