Scaling Effective Innovations

Implementation and scaling evidence-based practices and other effective innovations is a challenge in criminology as it is in all social sciences. Yet, scaling is essential to achieving socially significant outcomes envisioned in state and federal legislation and recommendations for reform (Charles Colson Task Force, 2016). In their article, Christopher J. Sullivan, Brandon C. Welsh, and Omeed S. Ilchi (2017, this issue) provide one of the few attempts to estimate the social impact of known variables related to an evidence-based program. Their computer simulation model holds promise for estimating outcomes of scaling in human service settings. The computer model is useful for evidence-based interventions that have substantial data on implementation influences (impact of high or low fidelity) and on intervention influences (impact on offense rates or recidivism). In this policy essay, we explore scaling more broadly with emphases on factors that should be considered, not just those that are available. Scaling is defined as the extent to which an innovation is used with good effect in the entire population of interest. It is argued that delivering the innovation as intended (with fidelity) and sustaining innovations in practice are essential to achieving social impact. Innovations refer to something new and previously unused by potential users or recipients (Rogers, 1995). Human services include all forms of organized prevention and intervention efforts in corrections, health, education, and social services (Herzberg, 2015). Implementation is defined as the supports required to purposefully and reliably produce full and effective uses of innovations in practice (Fixsen, Blase, Metz, and Van Dyke, 2015). Implementation capacity refers to the availability of implementation teams with the knowledge, skills, and abilities to develop competencies and affect change with practitioners, organizations, and systems. Social impact is defined as the product of effective innovations used with fidelity (the numerator) at a sufficient scale to produce noticeable benefits for the entire population of interest (the denominator) (Perl, 2011). In this conception, scaling innovations requires scaling implementation capacity. To illustrate this point, we compare atom-based innovations with interaction-based innovations and we highlight common components for producing and sustaining effective innovations at scale. Given the significant public and private investments in developing innovations that are intended to improve human services (Clancy, 2006), scaling the use of effective innovations in practice has global implications for the safety, health, and happiness of individuals in society. As pointed out by Sullivan et al. (2017), information is becoming available to predict scaling outcomes. This allows a move away from relying on the broad conception that scaling must always take into account a “discount” or “penalty” of unknown magnitude (Tommeraas and Ogden, 2016). First, good progress is being made in developing effective innovations in human services (Abrams et al. , 2012; National Institute of Corrections, 2004). The Nurse-Family Partnership (NFP) program cited by Sullivan et al. is an excellent example of an effective innovation. Second, key implementation factors related to the use of evidence-based programs and other innovations in typical practice settings are being operationalized and tested in implementation science (Glisson et al. , 2010; Metz et al. , 2014). There are now more than 30 implementation frameworks (Meyers, Durlak, and Wandersman, 2012; Tabak, Khoong, Chambers, and Brownson, 2012). With advances in research, frameworks are increasingly evidence based. For example, the active implementation frameworks (Fixsen et al. , 2015) are based on a thorough review of the implementation evaluation literature (Fixsen, Naoom, Blase, Friedman, and Wallace, 2005) and systematic examination of practice knowledge from experienced developers and implementers (Blase, Fixsen, Naoom, and Wallace, 2005). Finally, there is growing recognition that current systems produce current results and systems must change if new and improved results are to be realized (Clay-Williams, Nosrati, Cunningham, Hillman, and Braithwaite, 2014). Although work continues on developing innovations and improving and extending the science of implementation, attention is now being given to scaling human service innovations in communities across the nation and the globe to produce social impact. With scaling defined as the extent to which an innovation is used with good effect in the entire population of interest, only knowing how many individuals, organizations, or systems are benefiting from the use of an innovation means little as a scaling metric (Fixsen, 2009; Horner et al. , 2014; Spicer et al. , 2014). The denominator and the quality of the numerator both need to be defined and operationalized when determining scale and assessing social impact. The denominator for scaling is the entire population of intended beneficiaries. For example, the U. S. Bureau of Justice Statistics reported that more than 2 million adults were incarcerated in 2013 and nearly 5 million adults were on probation or parole. The Office of Juvenile Justice and Delinquency Prevention documented that, on average, more than 55, 000 delinquent youth were incarcerated in Colorado on any given day in 2014. The National Institute of Mental Health estimated that more than 10 million adults have one or more forms of severe mental illness in the United States. The World Bank is concerned with alleviating extreme poverty for greater than 1 billion people in 188 countries living on US1. 50 or less a day. In education, approximately 60 million students attend 98, 000 schools in the United States. An estimated 250, 000 people in the United States have been diagnosed with pulmonary hypertension, a rare and fatal disease. As these examples illustrate, the denominator for scaling varies greatly across domains, levels of systems, and specific populations of concern that could benefit. Scaling is essential to realizing the promised benefits of innovations in human services. Innovations cannot produce social impact unless they are used as intended in practice with the population of intended beneficiaries. Although identifying and quantifying the population of intended beneficiaries (the denominator) is straightforward, determining the numerator in human service applications is complicated. The numerator is the number of members of the population who experience the innovation (with fidelity) and, therefore, have the opportunity to benefit. The numerator for scaling is based on the number of recipients experiencing the innovation with fidelity. The quality of the innovation as it is delivered in practice operationally defines the numerator. Fidelity, the extent to which an innovation is used as intended, is an important determinant of the social benefit of innovations (Miller and Miller, 2015; Naleppa and Cagle, 2010). A case study of the importance of fidelity in scaling involves the inoculation of children with the Salk polio vaccine. Historically, some early batches of the newly developed Salk polio vaccine used by doctors to inoculate schoolchildren were substandard (low-fidelity production). Those substandard batches of polio vaccine increased the chances of children contracting the disease, which is not an outcome that benefits recipients or society as a whole and certainly not one to be scaled up. Those children who suffered the low-quality vaccine received an injection but did not receive a high-fidelity inoculation for polio. They would not count in the numerator related to scaling because they did not experience the intervention delivered as intended. Fortunately, medical personnel quickly detected and corrected the problem, and the damage to children, although severe for infected children, was limited for the population (Bayly, 1956). Assessments of fidelity in human services are not common, and their absence presents problems for calculating the numerator for scaling. The quality of an innovation as it is delivered in practice is at risk because human services are interaction based—they involve a human provider and a human recipient, each with unique personal characteristics and changing emotional states and circumstances, that influence one another. In many fields, innovations are atom based. For example, when developing improved highway bridges (McNichol, 2006) and new components for rockets (Boyle, 2003), the innovations are atom based, meaning they remain static after they are developed and do not change from one engineer to the next, shift into new configurations overnight, or decide not to show up. Even though they may be complex, developing and scaling atom-based innovations present “tame problems, ” whereas interaction-based innovations present “wicked problems” (Rittel and Webber, 1973). Given the complexity of multiple humans interacting with and influencing one another, wicked problems are difficult to define, “fight back” when solutions are attempted, and require continual resolution (re-solution) to maintain and improve impact. Wicked problems associated with interaction-based innovations impact the numerator for scaling because they impact quality/fidelity. When attempting to scale interaction-based innovations, fidelity is difficult to achieve and even more difficult to maintain as practitioners, supervisors/coaches, managers, and directors come and go in a sea of change. When attempting to scale interaction-based innovations, multiple challenges may add up to produce a negative effect that “is somewhat larger than the expected sum of the different factors” as indicated in the findings by Sullivan et al. (2017). This nonlinear effect further underscores the complexity of scaling up. Given the complexity of interaction-based innovations, the quality of the numerator can be difficult to assess. A good fidelity measure assesses the extent to which the essential components of an interaction-based innovation are present and used as intended (with fidelity) in daily practice (Schoenwald and Garland, 2013). In addition, a good fidelity measure is highly related to outcomes—the better the innovation is delivered in practice, the better the outcomes for users and recipients. With a strong relationship between the two, if fidelity is known, then outcomes can be predicted (Bond and Salyers, 2004). The complexity of interaction-based innovations also requires precise and persistent implementation capacity to produce high-fidelity uses of innovations reliably over time and across staff. For example, in the computer simulation run by Sullivan et al. (2017), arrest rates increased by 70% “when implementation factors are not fully considered. ” Implementation capacity is largely missing in human service systems (Schofield, 2004). The combination of lack of fidelity measures and the near absence of implementation capacity stand in the way of generating a numerator that meets and sustains a standard for quality (e. g. , high fidelity) needed to produce socially significant outcomes. Another impediment to scaling is sustainability. A numerator cannot continue to increase without attention to sustaining the quality of an innovation by those who already are using the innovation. Out of necessity, developers initially direct attention to expanding the number of new users (“adoption” of innovations). Over time, as the number of users increases, more attention must be paid to de-adoption and re-adoption (Massatti, Sweeney, Panzano, and Roth, 2008; McIntosh, Mercer, Nese, Strickland-Cohen, and Hoselton, 2015) while continuing to expand the numerator by developing new users. A container that is draining as fast as it is being filled simply maintains the same level. Likewise, if the number of users “de-adopting” an innovation equals the number of new users “adopting” an innovation, the numerator stays the same and scaling plateaus at that point. Any successful scaling attempt must attend to sustaining those who are already using an innovation while continuing to develop new users. Given that the denominator (the population of interest) is somewhat fixed, social impact depends on scaling the numerator. In atom-based business environments, the production methods (manufacturing processes) for an innovation are essential to scaling (increasing the numerator). For example, the initial production of newly invented silicon-based integrated circuits was problematic with approximately one usable silicon wafer produced out of 100 attempts. When the engineers examined the low-fidelity wafers, they discovered that hair, flakes of dry skin, pesticides in the air (they were in the middle of citrus groves before Silicon Valley paved them over), and so on contaminated the manufacturing process. This led to the development of ultra-clean environments for producing silicon wafers and for assembling computer hardware and other electronics (Kaeslin, 2008). With improvements in production processes, manufacturing plants could send usable (high-fidelity) atom-based products out the door at an increasingly rapid pace (Ohno, 1988). Scaling is done by increasing the size and number of ultra-clean production facilities nationally and globally. The job is not done when atom-based products leave the production plant. For sustained high-fidelity use, vaccines need to be stored properly, automobiles need to be serviced and repaired, computer software requires frequent updating (preferably automatically, to take users out of the process), and electronic parts are bound to fail under certain environmental conditions or after so many hours of use. Whole industries have been developed to maintain, repair, and improve atom-based products after they have been produced (sustain the numerator). For interaction-based innovations, the challenges involved in production and maintenance are compounded many times over. “Production” depends on the skill of an implementation team (Metz et al. , 2014) that does staff selection, training, coaching, and fidelity assessments in hospitable organization and system environments. “Maintenance” requires ongoing support from coaches and facilitative administrators and repeated assessments of fidelity and outcomes. Given the difficulties in reliably producing human competencies and their sensitivity to many “contaminants” in human service environments that are anything but “ultra-clean, ” the shelf life of many interaction-based innovations is not long. In this sense, human interaction skills are “perishable” with a “use by” date that is short. The development of implementation teams in human service organizations is tantamount to moving the production unit into the service environment (Metz et al. , 2014; Schofield, 2004). Interaction-based innovations without the ongoing support of implementation teams stand little chance of high-fidelity use or sustained use in turbulent human service environments (Brunk, Chapman, and Schoenwald, 2014; Fixsen, Blase, Timbers, and Wolf, 2001; Tommeraas and Ogden, 2016). If a human service system already had implementation capacity in the form of skilled implementation teams that were integrated and aligned to support the full and effective use of innovations, the social impact of effective innovations already would be at a high level and improving every year. Unfortunately, that is not the case (Guerino, Harrison, and Sabol, 2011; National Center for Education Statistics, 2013; Starfield, 2000). Any approach to scaling must (from the beginning) include plans to initiate and manage necessary system changes to increase implementation capacity and sustain high fidelity use of effective innovations. Human service systems are legacy systems (Ulrich, 2002) made up of an accumulation of fragments of past mandates, good ideas, beliefs, and ways of work that evolved over many decades as legislators, leaders, and staff came and went. Legacy systems typically are fragmented, siloed, and inefficient. To realize social impact, organizations and systems need to be designed on purpose to produce and sustain high-fidelity use of effective innovations. This requires changes in legacy systems, often in dramatic and transformative ways (Osborne and Gaebler, 1992), to produce, implement, sustain, and improve innovations. Transformation of systems may require only small changes that make a big difference (Lanham et al. , 2013). For example, Ulrich (2002) found that transformed legacy systems retained approximately 80% of the original system. Only 20% had to change to produce dramatically improved performance. Nonetheless, distinguishing the 20% from the 80% presents problems when change must occur in human service systems that continue to function as-is during the time of transition. “Transformation zone” strategies and recursive practice-policy communication methods have been developed to discover the 20% that needs to change and then changing it in human service systems (Fixsen, Blase, and Van Dyke 2012; Ulrich, 2002). In the scaling and system change literature, there is debate among researchers about the merits of top-down or bottom-up approaches (Klingner, Boardman, and McMaster, 2013; Pülzl and 2006), the of fidelity or in the use of innovations et al. , 2012; and and the of or strategies and 2011; 2008). There also is of scaling strategies that more and approaches and or complexity that forms and with (Lanham et al. , 2013; 2008). In there is in scaling there is only top-down and bottom-up capacity development involves and strategies fidelity and are A approach to the work to be done and complexity to make of these and the of the The need for change is in an example from global and the of the vaccine and the global efforts that in and led to the of in The global and of in is an example of a approach to implementation and scaling. et al. and how the global had to and quickly related to the quality and of the of and personnel in and and among and populations as a of and approaches may the job but they are not on change and do not leave improved implementation capacity in their but did not or sustain improved global or systems with the capacity to other or in the decades (e. g. , the that in the The computer simulation model by Sullivan et al. has the potential to investments in scaling effective innovations in criminology and human services. Given the importance of the simulation model holds promise for policy and A simulation approach is useful when it from of the of the innovation, and the relationship between fidelity and of the impact of implementation variables (e. g. , staff selection, training, and organization and on fidelity of the use of the and of the influence of factors for support for and coaching, and impact of that may or use of effective implementation methods and use of an effective innovation. As in this policy essay, scaling innovations in human services can be operationalized and social impact can be in of the quality of innovations as they are used in the denominator is a good to The of scaling is in the to produce a numerator substantial to realize population The quality of the innovation as delivered in practice on use of effective implementation use of an effective innovation depends on scaling implementation capacity to sustain, and improve of the innovation. It little to in attempts to scale innovations that do not produce intended have the and we now are using and we are using or policy a and an to take of are not the same as using an innovation as intended (with producing promised outcomes in typical practice and sustaining the innovations and implementation supports so they the new standard practices in human service systems. to scale or innovations are a of time, and Yet, in strategies that do not work is the and not the as and for solutions in human service systems that are to in ways that produce intended outcomes and Institute of 2001; et al. , and U. S. of Education Institute of Education 2010; U. S. of Health and Human administrators and simply do not have the for change available to It is a who only to the to a The means are not available to achieve the in innovations and implementation strategies produce the of innovations and new often in human service settings. It is time for change in human services. to develop innovations is and the number of evidence-based innovations is we to do in The availability of innovations that are effective but not used in practice has attention on implementation With rapid advances in implementation we are to make effective use of innovations in typical human service organizations and systems and 2005). the is to scale implementation capacity in the form of skilled implementation teams to produce on purpose social impact for the effective innovations at as as those to be of nation and global is of the National Implementation and of the Implementation In Implementation A of the and has been active in developing implementation science as a to was of the that developed and one of the few evidence-based programs for in systems. is of the National Implementation and at the of at Implementation A of the and led efforts to and implementation and scaling methods in human services. was of the and of the during the when each was for effective is the of implementation science at in with a on scaling The and The Partnership programs in With a in social work research, and implementation interest in how human and services can be and scaled up in