Implementation Science for the US President's Emergency Plan for AIDS Relief (PEPFAR)

Working with implementing organizations and governments in over 32 countries, the US President's Emergency Plan for AIDS Relief (PEPFAR) has contributed to the rapid acceleration of HIV treatment access, availability of care and support services, and HIV prevention interventions. In the first phase of PEPFAR, these activities were appropriately carried out in an emergency fashion with the goal of using available interventions to reduce mortality and alleviate suffering from HIV disease as quickly and effectively as possible. Many lessons have been learned through examination of programs, including simple evaluations and operations research. Commensurate with the emergency response, however, state-of-the-art monitoring, evaluation, and research methodologies were not fully integrated or systematically performed. In the second phase of PEPFAR, characterized by an increased emphasis on sustainability, programs must demonstrate value and impact to be prioritized within complex and resource-constrained environments. In this context, there is a greater demand to causally attribute outcomes to programs. Better attribution can be used to inform midcourse corrections in the scale-up of new interventions (eg, male circumcision) or to re-evaluate investments in programs for which impact is less clear. To meet these demands, PEPFAR is adopting an implementation science (IS) framework to improve the development and effectiveness of its programs at all levels. IS is the study of methods to improve the uptake, implementation, and translation of research findings into routine and common practices (the “know-do” or “evidence to program” gap).1,2 For example, IS was used to evaluate the routine operational effectiveness of the South African National Prevention of Mother-to-Child Transmission Programme.3 Investigators explored the survival of HIV-free infants across program sites and identified specific sources of variation such as health system factors (eg, limited antenatal visits and lack of syphilis screening) and individual behaviors (eg, breastfeeding practices). By framing the problem through IS, the study revealed opportunities for improving program performance that could be translated into immediate solutions (eg, improving quality of care, infant feeding counseling). In this way, IS proved to be a valuable tool that was used not only to improve program effectiveness, but also to explain what worked, why, and under what circumstances. Although no less rigorous than biomedical research dictated by a static protocol with robust internal validity (ie, “proof-of-concept” research with a precisely defined and narrow objective), an IS approach represents a paradigmatic shift in emphasis to greater external validity. The IS scope is also broader, seeking to improve program effectiveness and optimize efficiency, including the effective transfer of interventions from one setting to another.1,4 The methods of IS facilitate making evidence-based choices between competing or combined interventions and improving the delivery of effective and cost-effective programs. Development of a PEPFAR Implementation Science Framework Components of the Implementation Science Framework Building on a successful experience with some elements of IS in the first phase of PEPFAR, the Office of the US Global AIDS Coordinator, in consultation with its US Government PEPFAR implementing agencies and a broad group of academic, programmatic, and methodological experts, is in the process of developing an IS framework. The framework will provide structure, methodological rigor and diversity as well as knowledge generation to meet the needs of the program and the global community. This framework, described here, incorporates the fundamental components of IS: monitoring and evaluation, operations research, and impact evaluation (including modeling and cost-effectiveness analyses). Monitoring and Evaluation Monitoring is the routine, daily assessment of ongoing activities, inputs, outputs, and progress.5 By contrast, evaluation assesses what has been achieved (Table 1).5 Using prevention of mother-to-child transmission (PMTCT) as an example, monitoring and evaluation (M however, all eligible sites eventually receive the program, ensuring equity. In addition to randomization, the impact evaluation tool kit includes quasiexperimental methods that generate a valid counterfactual without random allocation through use of statistical methods that permit causal attribution of outcomes to the program.15,16 Modeling can also be used to integrate multiple data sources and permit inferences about impact, especially when empiric data on outcomes are limited or nonexistent.21,22 Because such methods often draw on existing data such as demographic surveys, they allow for cross-sectional or retrospective evaluations, and they are often quicker and cheaper than experimental designs. However, they can be more analytically intensive and have a greater chance of suffering from both selection bias and violation of statistical assumptions than do experimental designs.15 There are numerous examples of successful impact evaluations.17,23,24 For example, stepped wedge designs have been used effectively in trials assessing the impact of hepatitis B vaccination,25 nevirapine to prevent mother-to-child transmission,26 education programs to improve adherence to HIV treatment,27 and tuberculosis screening and treatment in HIV-positive men.28 In addition, the innovative methods of adaptive clinical trial designs in combination with such approaches hold promise as a way to assess multicomponent, combination HIV prevention programs.29 Application of the Implementation Science Framework to PEPFAR Programs and Research Applying an IS framework to PEPFAR programs will sharpen our ability to support partner countries in choosing strategic programs that provide the most benefit using the most efficient methods. To accomplish these goals, the entire IS continuum encompassing monitoring and evaluation, operations research, and impact evaluation should be unified into a coherent framework, in which results across the full spectrum can support successful implementation of critical programs. All PEPFAR programs must be subject to the most rigorous IS methods and evaluations that are appropriate and feasible. Impact evaluations should be prioritized for programs of unknown or uncertain efficacy, whereas all programs should be evaluated for opportunities to improve efficiencies through operations research. Similarly, M&E should be incorporated into all programs to determine the adequacy of program implementation, coverage, and outcomes. Existing and Future PEPFAR Programs and Research IS-based evaluations must be added as needed to existing programs, especially where there are questions about effectiveness or efficiency. For these programs, impact evaluations may be possible through slight changes in the program (eg, randomly adding an incentive program to increase testing or adherence) and/or additional data collection to create a de novo baseline with clear outcomes. In all cases, operations research can be used to improve program efficiency, and M&E can monitor whether programs are implemented as intended. It will also be essential for new programs to include evaluation plans designed and implemented in tandem with the program. Designs of new programs should consider the complete array of IS tools: rigorous impact evaluations (eg, stepped wedge evaluation designs incorporated into program roll-out), multiple models of service delivery and/or implementation to be explored though operations research, and establishing a system to routinely monitor the program's inputs, outputs, and outcomes. As discussed, an IS approach incorporates strategic use of real-time data collection that permits ongoing program corrections. Programs can be improved on an ongoing basis using input from a variety of stakeholders and data collected from multiple levels of implementation: from inputs to impact. To gain further input into the adoption of the IS framework, the Office of the US Global AIDS Coordinator is forming a Scientific Advisory Board, composed of leading academic, nongovernmental and US Government researchers, that will help shape IS priorities to maximize PEPFAR impact (potential examples are listed in Table 4).TABLE 4: Sample Potential Implementation Science Questions for PEPFARCONCLUSION PEPFAR has unequivocally had enormous beneficial impact. The scale and urgency of the program as developed were commensurate with the scale and urgency of the epidemic and resulted in millions of lives saved and infections averted. In the next phase of PEPFAR, emphasis must also be placed on the development and contribution of knowledge about HIV/AIDS program implementation to the global community. An IS framework will permit identification of high-priority implementation questions and development of tools with which to answer them. Acceleration of this work will permit PEPFAR to support strategic interventions and focus them where they will have the most impact while simultaneously improving implementation efficiency and sustainability. ACKNOWLEDGMENTS We thank Drs Stefano Bertozzi, Nicholas Jewell, Margaret Brandeau, and Brian Lewis for helpful critiques of an early draft of this paper.