Evaluation of Interventions Designed to Prevent and Control Injuries

Violent and unintentional injuries place a severe physical, emotional, and financial burden on US communities. Injuries affect people of all ages, from infants through older adults, and are the leading cause of death in the first four decades of life (1). In 1995, the economic cost of injuries was estimated to be over $260 billion, including the costs of health care and lost productivity (2). Despite the large burden on US communities, injury prevention is still a relatively new area of public health. In 1992, Congress mandated the establishment of the Centers for Disease Control and Prevention’s National Center for Injury Prevention and Control to coordinate research and programmatic responses to the problem of injuries in this country (3). In partnership with other federal, state, and local organizations and universities, the Center has encouraged the use of a population-based or public health approach to injury research and programs. This approach complements engineering, mental health, and criminal justice approaches to injury prevention and control. Evaluation of interventions is an important aspect of injury prevention. Results of evaluations have the power to change injury prevention practice. For example, bicycle helmet use increased following demonstration of the impact of helmets on brain injury (4–6). The number of programs promoting installation of smoke detectors and provision of fire injury prevention education grew once these programs were shown to be highly effective and to save money (7, 8). The number of states with laws prohibiting blood alcohol concentrations of 0.08 percent or higher increased after such laws were shown to decrease fatal motor vehicle injuries (9–11). In this paper, our goal is to discuss the evaluation of injury prevention interventions in two contexts—intervention research (efficacy or effectiveness trials) and program evaluation. Our goal is not to present an exhaustive methodological review of evaluation strategies. The reader is referred to several excellent articles and texts that provide overviews of evaluation methods used in research and program evaluation (12–17). Rather, our intent is to review selected broad issues and challenges faced by injury researchers and practitioners as they undertake an evaluation activity. Attention to these issues is important to ensure that evaluation findings are both credible and useful. Evaluation, to borrow the definition from The American Heritage Dictionary of the English Language (18), is action “to ascertain or fix the value or worth of” something, in this case injury prevention interventions. Scriven provides a well-known definition in the field of program evaluation, namely, “the process of determining the merit, worth, or value of something or the product of that process” (19, p. 139). Evaluation is ultimately about making judgments—judgments by scientists about whether an intervention is effective in reducing injury risk and judgments by practitioners about whether an intervention should be continued, modified, or eliminated from an injury prevention program. Evaluation is the overarching guide that helps to assure effective and efficient use of limited resources. In this paper, we artificially divide the overall evaluation process into two areas: intervention research and program evaluation. “Intervention” refers to an activity or device employed with the goal of reducing or preventing injuries. Interventions can include medical activities, for example, acute care of the injured in emergency departments or rehabilitation care for patients with traumatic brain injury. In this paper, we emphasize nonmedical interventions aimed at primary prevention of injuries. These interventions can be of many kinds: behavioral (brief motivational interviews in emergency departments to reduce problem drinking and driving (20)), social (family therapy interventions for youth at risk for violent behavior and institutionalization (21)), community (multilevel intervention to reduce alcohol-related motor vehicle crashes and assaults (22)), policy (passage of 0.08 blood alcohol concentration laws (9)), environmental (traffic calming measures (23)), or technologic or biomechanical (hip pads for preventing fractures in the frail elderly (24)). We define intervention research as systematic investigations conducted to create generalizable knowledge about effective injury prevention interventions (25). Intervention research is typically aimed at evaluating new and innovative strategies (efficacy trials (20, 24, 26, 27)) or testing the application of proven interventions in a setting or population significantly different from that in which they were originally tested (effectiveness trials (21, 26, 27)). As we note below, intervention research is typically more researcher-driven and, as conditions allow, uses rigorous research methods. Program evaluations are usually practitioner-driven, providing credible information on whether to implement, improve, continue, or expand a specific intervention program rather than generating knowledge that is generalizable to other situations. For example, Luria et al. (28) evaluated Safety City, a program designed to increase children’s safety knowledge and behaviors that was implemented in Columbus, Ohio, city schools in 1989. Evaluation findings showed that the program as it was currently being practiced was not successful in changing children’s knowledge about safety procedures for crossing streets, dialing 911, and avoiding strangers. The sponsoring agency made changes in the program based on the evaluation, including ensuring that the curriculum used age-appropriate language, providing more specific information on teacher activities, and providing a letter to parents highlighting parental strategies for encouraging child safety behavior. Because of a variety of social and economic circumstances, program evaluation activities may not attain the level of research rigor seen in intervention research. Although the lines between intervention research and program evaluation are not always clear, in general intervention research will involve a more complex design and require more data collection. Whereas control and comparison groups are critical to research, time and financial limitations often limit the use of these groups in program evaluations. Considering the viewpoints of various stakeholders, such as program managers, staff, and clients, in developing evaluation questions is a cornerstone of program evaluation. Intervention research questions may be driven by identified gaps in knowledge and by the needs of public agencies that fund intervention research. Evaluations may be designed to answer several kinds of questions related to the process, outcome, cost, or impact of interventions. In process evaluations, details are gathered on intervention implementation and quality. Outcome evaluations assess changes in proximal indicators, for example, attitudes, beliefs, and behaviors, or environmental and policy changes that are believed to be related to reductions in injuries and deaths. Evaluations of the cost or cost-effectiveness of injury-related interventions have been infrequently conducted. Such evaluations describe the effectiveness of a particular intervention in the context of the costs of implementation, the costs to study participants, and the costs associated with any side effects of the intervention (8, 29–32). Legislators are heavily influenced by cost-benefit data in setting budget priorities (33). Impact evaluations assess changes in the frequency or severity of injuries and deaths resulting from a particular intervention. In deciding which interventions should be further studied or implemented more widely, public health practitioners and policy makers often ask for data on impact or cost-effectiveness. However, there are many circumstances in which impact data may be difficult to obtain. When the impact one would want to measure is rare (e.g., the killing of a child by a teenage parent) or the impact of the intervention might be difficult to disentangle from the impacts of other concurrent changes (e.g., changes in the economy), it may be very important to measure proximal changes (e.g., improved parent-child interaction). Potential problems with the use of proximal measures as evidence of intervention effectiveness are discussed below. The acceptance of more proximal changes may be greater in circumstances where it is considered unethical not to intervene or where failure to conduct an intervention would have an apparently negative outcome. A series of issues should be considered in all evaluations, regardless of the context in which the evaluations are conducted. The Centers for Disease Control and Prevention’s “Framework for Program Evaluation in Public Health,” published in 1997, serves as one foundation for our discussion (34–36). This document describes six activities that should be considered in conducting any program evaluation in public health. Although the Framework focuses on program evaluation, many of these activities are important in conducting any type of evaluation. It is best to determine the evaluation design before the intervention begins. Once an intervention has started, the opportunity to collect critical baseline data or early process data may be lost if these factors were not built into the evaluation design. Furthermore, if control or comparison groups are not selected before initiation of the intervention, it may be impossible to collect data that will allow appropriate comparison. With research studies, the unit of analysis (e.g., the individual, school, or community) must be identified early to ensure that there is sufficient power (e.g., sample size) to draw conclusions from the study. In addition, developing the design before the intervention begins allows stakeholders to be engaged early in the process. This often leads to a better evaluation and enhances the likelihood that stakeholders will use the findings. Not infrequently, program evaluations cannot be initiated until after intervention implementation has begun. Despite the loss of some opportunities, important evaluation questions can still be answered. Intervention participants can provide information on program benefits and satisfaction, and surveillance data can be used to assess possible impacts. The Centers for Disease Control and Prevention’s program evaluation framework identifies three groups of potential stakeholders for program evaluations: persons involved in program operations, persons who are served or affected by the program, and primary users of the evaluation (34). Within these groups may be clients, funders, policy makers, program managers, researchers, and other persons who are positively or negatively, directly or indirectly affected by the intervention. Evaluation activities are key to building high-quality injury prevention programs. Key to conducting high-quality, useful evaluations is between the and community stakeholders, injury program the level of will to the and of the evaluation, should in a the resulting product is often between researchers and stakeholders are often initiated the evaluation begins. These can be it is through that researchers and injury program a level of and of and that can the evaluation different for stakeholders in program evaluations These from to stakeholders involved in several of the evaluation (e.g., or to evaluations, in which scientists more as with the of the evaluation being by stakeholders of community stakeholders may not be as in some example, in the evaluation of such as pads designed to reduce injuries related to However, in this is at some to ensure that the device or other intervention is to or or program evaluations of interventions that are conducted in always include at various For example, in research studies, community may with scientists in generating study the intervention, participants, and In a example, et al. from several public and community agencies used a to conduct a community of two interventions aimed at reducing early and This approach in a of such that the the for community and study activities in a of intervention activities is more to after of the study if community are engaged in the research The of the intervention to be evaluated should be as should the of the intervention to risk and this an it is an intervention is the for to the there is a between the risk and factors and the intervention, the intervention is to be can be useful for the intervention and measures describe the risk and intervention and describe both and including and based on data from a variety of such as or other research and Once the intervention has been the of implementation and should be to should not be encouraged to interventions. Although interventions should be for the or of the population of helps ensure that intervention or that are critical to in interventions it difficult to an intervention or an to a particular intervention. Once the evaluation questions have been and the intervention the are to the evaluation design and the to be researchers are to conduct trials that through of or communities. The level of must the intervention and the unit of For example, if the intervention an should be to the or control with evaluating between as a are in effectiveness trials and program evaluations and are used is not and baseline between groups can be in other (e.g., evaluation of a bicycle helmet and an evaluation of changes in bicycle helmet laws and an are two of evaluations In both studies, interventions were and of was not some of the and about and some approaches to evaluation have been For example, and describe the in which the of an intervention are on the of intervention and the the evaluation data the data on the of the and and the data are with that the and were can be This is to have more for than for evaluation activities on to measure such as and interviews and case may be useful for conducting process evaluations, or evaluation For example, in a study unintentional injury prevention programs for with programs by health care measures showed that intervention in changes in attitudes, or behavioral in comparison with a control However, questions on the into possible effects of the such as increased before and of the possible of behavior study important information on on the effects of in interventions on the behavior as as are excellent for and data are in which interventions are very difficult to For example, in the case of the effects of the various is difficult a very and complex evaluation design In effectiveness research, there is a to interventions as a in where many of the have been tested in trials or where it is believed that intervention may be In some data for other such as medical data or surveillance can be in program evaluation. This is for evaluations of policy where data from several time before policy implementation may that the of the of for many or and or after evaluation of the impact of the of both and or surveillance in such In this data from control in unintentional deaths and by after new were into place A review of data on associated with the unintentional of for showed that percent from the before policy implementation through evaluations of prevention programs have used data to assess program surveillance including health showed a in following initiation of prevention programs and used data on for the to prevention programs that were in the and it is not possible to a about cause and in these such data the that a Evaluation may one of several different of control or comparison control or comparison groups are always used in and effectiveness The use of an control comparison of one that has an intervention with that in intervention. When comparison rather than control groups are it is possible to the effects of two or more interventions. As an to an control or comparison study participants may as through the use of a For example, research has shown that a design can provide This design study participants after the intervention about attitudes, or behaviors and attitudes, or behaviors to a specific in time before the intervention In program evaluations, control or comparison groups are used often of and other et al. several methodological challenges they faced in evaluating a and injury prevention program. For example, many that to as a control was a of and were about the of the In to these these researchers the and to be with one control used to assess intervention effects on the evaluation questions and the for The evaluation and impact data before and after the intervention is may provide important information on and in change should always be in the between proximal and For example, et al. used a design to the of a intervention and installation of safety on reductions in the number of environmental in the and on the frequency of older the intervention in reductions in four of the the of was not In other measures may the effectiveness of interventions that will ultimately be shown to be effective This may the effects of an intervention are or interventions to change social The of data an evaluation is critical to outcome. In the case of or it is often best to measures that have been used and for which there are data (e.g., or For other such as of injury or care must be to data with As in social and environmental of related to these such as social social community and and are greater We the reader to several texts and articles for of and critical design is the of the number of to conduct and after the intervention is the design has been the However, made and after intervention implementation can provide data on three important in policy implementation or environmental effects of the and of effects after the intervention is and can provide useful information for in the of an control the limited for evaluations usually in a between the frequency of data and the and of data of intervention effects the implementation is a key for public health interventions that effects are over if for the cost made over time provide information on whether the intervention has a or whether the is over In an evaluation of a fire prevention program and fire were conducted and after the program installation and of the over injuries in the In a series of evaluating the impact of and conducted by over a intervention were conducted the first of the and at The intervention significantly to the of including of child and and injuries. intervention and of and of alcohol are a number of important in conducting intervention evaluations in research or programs. conducting by the US of and are with the the of in of the of (25). that researchers the potential costs or and benefits of in research and that these be in the research, that systematic investigations conducted to create generalizable must from an review and health and social organizations have that for research In to research, program evaluations that are conducted to an intervention program are considered public health and not usually require (25). on whether they require for program evaluations, and they should be before the evaluation activity begins. However, regardless of practitioners conducting program evaluations should use and other strategies for the of to ensure that the and benefits of the activities and are not by that must be considered in evaluating interventions is the type of intervention to be to control that control should not be something of potential or that they should not be the best intervention in community In a evaluation of a intervention risk behaviors, that all persons in in control should be the opportunity to behavioral which were the best proven intervention in the local at that time In research studies, the the conduct of research at a particular or can provide on the of interventions to be to control In the case of program evaluations, at are as if such A is the to which injury researchers or practitioners are to provide the intervention to the control if that intervention is to be In once a medical or has been control are the In a evaluation of an intervention to increase bicycle helmet use schools were to the or the control Evaluation showed that the intervention been and control schools were the of the helmet program to many research provide to the of intervention activities in schools or communities. When community stakeholders have been engaged in the intervention research or program evaluation, local agencies or health departments may be to the effective intervention to once the research is intervention implementation and evaluation, it is not to have including some that are or For example, in review of youth studies, et al. to the of in interventions in groups can increase problem behaviors (e.g., and However, implementation can et al. that of laws in increased for other must care not to on a that they to other and or to determine programs or not The US on and negative side effects in of the effectiveness of interventions. the has that the and of side effects is in the For example, of interventions encouraging the use of safety and interventions for reducing driving evidence of side effects resulting from the interventions. It is whether the problem is an of of or and of such with various stakeholders can be in and intervention that are or interviews of study participants can provide on the benefits and problems associated with in the intervention. from an intervention research or program evaluation must be with stakeholders in a with stakeholders often of and about interventions to be effective in the school, or other and practitioners have an to they can to the use of data in public health programs typically not evaluation in This is the case with in which there are However, researchers and practitioners have an to or negative findings in to the of injury prevention and control they have a to in where practitioners will have an opportunity to to the intervention. of often from including details on the intervention, and and other information to conduct the intervention. for of these of data include and with these details can a to or use an intervention shown to be several groups conducting systematic review such as the and the of the to have information about the or effectiveness of injury interventions. These allow more than can often be made on the of For example, evaluations of the have that evidence a variety of of motor vehicle the use of programs to child and the use of programs to youth As is in all public health evaluations of injury-related prevention interventions many These include issues related to the of injury control to interventions other public health and changing social and other issues that affect of injury. of financial is often the evaluations are not or are Public health responses to the injury problem can be as they are in the context of the of the problem (1). the for public health designed to and control injuries would in an increase in for evaluation, in in increased policy makers of programs. As we injury interventions may have an impact on other health and of injury or When injury prevention should include For example, programs such as that are of prevention programs for the elderly may to be at as to and mental health as to injury The impact of changes may increase as the of evaluations For example, large in deaths the between and and affected both and Such may it difficult to measure the impact of an intervention, if that impact is relatively to social of health over better and control of these In the field of public health, the of evidence as a for interventions to has been However, there to be a large number of injury interventions that are used have been The program was to criminal activity by to by in or at risk for criminal behavior. The intervention was highly for until an evaluation showed that it was an of that of used interventions that have not been evaluated include intervention parental programs designed to reduce unintentional and violent injuries to and emergency medical Public health in whether they use of interventions that have not been evaluated have or with we discussed to the However, there are a number of of that to be from trials from and other study The of the that therapy be used to in this has the level and type of evidence to determine the effectiveness of an intervention in the developing have the in the and program evaluation approaches in the and have for the of and of data used in evaluations. As we a number of injury intervention evaluations not to approaches to evaluation are as are methods that allow to use better (e.g., better to control for factors in of should be for from community and intervention studies, has been used in the to Despite the of conducting evaluations of health we as public health have a to our best to ensure that interventions we a We have a to conduct the best evaluations we into the kinds of issues in this