1. Implementation
Research:
A Synthesis
of the
Literature Dean L. Fixsen
Sandra F. Naoom
Karen A. Blase
Dean L. Fixsen Robert M. Friedman
Frances Wallace
Sandra F. Naoom
Karen A. Blase
Robert M. Friedman
Frances Wallace
Tampa, Florida
2005
3. Implementation Research: A Synthesis of the Literature
Contents
Chapter 1 Introduction ................................................................................................................... 1
Review Methods .............................................................................................................3
An Implementation Headset ...........................................................................................4
Implementation Defined ................................................................................................5
Degrees of Implementation.............................................................................................6
Chapter 2 Implementation in the Context of Community ................................................................. 7
Research on Community Context...................................................................................8
Measuring Readiness.......................................................................................................9
Chapter 3 A Conceptual View of Implementation ............................................................................11
Conceptual Framework.................................................................................................12
Purveyors ......................................................................................................................14
Stages of Implementation Defined ................................................................................15
Exploration and Adoption .....................................................................................15
Program Installation ..............................................................................................16
Initial Implementation ..........................................................................................16
Full Operation.......................................................................................................16
Innovation .............................................................................................................17
Sustainability .........................................................................................................17
What We Know about Implementation Stages..............................................................18
Stages of Implementation and the Literature .........................................................18
Research on Stages of Implementation...................................................................18
Experimental Analyses of Implementation Strategies.....................................................20
Experimental Research: Ineffective Implementation Strategies ...............................20
Experimental Research: Effective Implementation Strategies..................................21
Chapter 4 Core Implementation Components .................................................................................23
Core Components Defined ...........................................................................................24
Core Components for Interventions .............................................................................24
Evidence-Based Practices and Evidence-Based Programs .......................................26
Implementing Practices within Organizations .......................................................27
Core Components for Implementation .........................................................................28
Overview and Definitions......................................................................................28
Sources of Core Implementation Components ......................................................31
Developing Self-Sustaining Implementation Sites .........................................................32
National Implementation Efforts ..................................................................................34
Chapter 5 Research on Core Implementation Components ..............................................................35
Staff Selection ...............................................................................................................36
Experimental Research on Selection ......................................................................36
Practitioner Selection: Additional Evidence ...........................................................37
Organization Staff Selection: Additional Evidence .................................................38
Purveyor Staff Selection: Additional Evidence ........................................................39
Staff Selection Summary ........................................................................................39
— iii —
4. Implementation Research: A Synthesis of the Literature
Staff Training ................................................................................................................39
Factors that Impact Training ..................................................................................40
Experimental Research on Training Outcomes ......................................................40
Experimental Research on Training Methods .........................................................41
Training Practitioners: Additional Evidence ...........................................................41
Training Organizational Staff: Additional Evidence ...............................................43
Staff Training Summary .........................................................................................43
Staff Coaching ..............................................................................................................44
Factors that Impact Coaching ................................................................................45
Experimental Research on Coaching .....................................................................46
Additional Evidence for Coaching .........................................................................46
Staff Coaching Summary .......................................................................................47
Evaluation and Fidelity .................................................................................................47
Staff Evaluation for Performance Improvement .....................................................48
Organization-Level Fidelity Assessments ...............................................................51
Factors that Impact Staff Evaluation for Performance Improvement ......................53
Experimental Research on Evaluation ....................................................................53
Staff Evaluation for Performance Improvement: Additional Evidence ....................53
Organization-Level Fidelity Assessments: Additional Evidence ..............................54
Staff Evaluation to Measure Adherence to Research Protocols ................................54
Evaluation and Fidelity Summary ..........................................................................55
Chapter 6 Organizational Context and External Influences ..............................................................57
Literature Related to Organizational Components and External Influence ....................60
Influence Factors at Work ......................................................................................62
Organizational Change and Development ....................................................................64
Evaluations of Core Organizational Components ..................................................65
Summary .....................................................................................................................66
Chapter 7 Conclusions and Recommendations ................................................................................67
Findings & Conclusions ...............................................................................................70
Implementation and the Status Quo .............................................................................71
Recommendations ........................................................................................................72
Recommendations for State and National Policy Makers .......................................72
Recommendations for Research on Implementation ..............................................74
Recommendations for Effectiveness Research on Practices and Programs...............76
Recommendations for Purveyors of Well-defined Practices and Programs ..............77
— iv —
5. Implementation Research: A Synthesis of the Literature
Appendices
Appendix A Review Methods .......................................................................................80
Appendix B W.T. Grant Project Literature Review Codebook ......................................84
Appendix C Experimental Studies ...............................................................................91
Appendix D Hypotheses for Advancing Implementation Science .................................95
References ..................................................................................................................................101
List of Figures
Figure 1 A Conceptual Framework for Implementation
of Defined Practices and Programs .......................................................................12
Figure 2 Implementation Framework Applied to Developing
Evidence-based Intervention Practices within Organizations. ...............................28
Figure 3 Core Implementation Components ....................................................................29
Figure 4 Implementation Framework Applied to Developing Self-sustaining
Implementation Sites within Organizations in Communities ...............................33
Figure 5 Multilevel Influences on Successful Implementation ............................................59
List of Tables
Table 1 A Summary of a Meta-analysis of the Effects of Training
and Coaching on Teachers’ Implementation in the Classroom .............................30
Table 2 Examples of Different Types of Fidelity Measures Across Programs ......................49
Table 3 Postulated Relationships Among Core Implementation Components,
Organizational Components, and External Influence Factors
that may Help Explain Various Implementation Outcomes .................................59
Table 4 Factors Deemed to be Critical to the Operation
of a Residential Treatment Program......................................................................66
Table 5 The Interaction of Intervention Effectiveness
and Implementation Effectiveness. .......................................................................69
—v—
6. Implementation Research: A Synthesis of the Literature
Preface
Over the past decade, the science related ing about, studying and promoting implementa-
to developing and identifying “evidence-based tion in human services.
About the Review practices and programs” has improved—how- For example, it became evident that thought-
This monograph summarizes ever the science related to implementing these ful and effective implementation strategies at
findings from the review of programs with fidelity and good outcomes for multiple levels are essential to any systematic
the research literature on consumers lags far behind. As a field, we have attempt to use the products of science to improve
implementation. The review discovered that all the paper in file cabinets plus the lives of children, families, and adults. That is,
process began by identifying all the manuals on the shelves do not equal real- implementation is synonymous with coordinated
literature reporting any efforts world transformation of human service systems change at system, organization, program, and
to collect data on attempts to through innovative practice. While paperwork practice levels. In a fundamental sense, implemen-
implement practices or programs and manuals do represent what is known about tation appears most successful when:
in any domain, including effective interventions, these tools are not being • carefully selected practitioners receive co-
agriculture, business, child used effectively to achieve behavioral health ordinated training, coaching, and frequent
welfare, engineering, health, outcomes for children, families, and adults performance assessments;
juvenile justice, manufacturing, nationally. Clearly, state and national policies • organizations provide the infrastructure
medicine, mental health, nursing aimed at improving human services require more necessary for timely training, skillful supervi-
and social services. effective and efficient methods to translate policy sion and coaching, and regular process and
Nearly 2,000 citations were found, mandates for effective programs into the actions outcome evaluations;
1,054 met the criteria for inclusion that will realize them.
in the review, and 743 remained • communities and consumers are fully involved
To this end, our intent is to describe the
after a full text review. There in the selection and evaluation of programs
current state of the science of implementation,
were 377 out of 743 citations and practices; and
and identify what it will take to transmit innova-
deemed to be most relevant, tive programs and practices to mental health, • state and federal funding avenues, policies, and
and 22 studies that employed social services, juvenile justice, education, early regulations create a hospitable environment
an experimental analysis of childhood education, employment services, and for implementation and program operations.
implementation factors. substance abuse prevention and treatment. The It also appears that relevant implementation
content is distilled from a far-reaching review factors and processes are common across domains
of existing implementation literature that looks (e.g., mental health, juvenile justice, education,
beyond the world of human services to organize child welfare). If this is true, then efforts to im-
and synthesize critical lessons from agriculture, prove the science and practice of implementation
business, engineering, medicine, manufacturing, have the potential for positive broad scale impacts
and marketing. As you will find, authors from on human services, across service systems.
around the globe share the rigors of attempting to In summary, the results of this literature re-
implement practices and programs and agree that view and synthesis confirm that systematic imple-
the challenges and complexities of implementa- mentation practices are essential to any national
tion far outweigh the efforts of developing the attempt to use the products of science—such as
practices and programs themselves. evidence-based programs—to improve the lives
During the course of the overall literature of its citizens. Consequently, a concerted national
review, select studies featuring robust experimen- effort to improve the science and the practice of
tal analyses of implementation factors also were implementation must accompany support for the
mined for common themes and definitions. As a science of intervention. The components of imple-
product of this work, conceptual frameworks and mentation and factors promoting its effectiveness
a corresponding lexicon emerged to help sum- must be understood, and we hope the frameworks
marize the information, create understanding, and recommendations introduced in this volume
and evolve testable hypotheses. Accordingly, this provide a foundation for this understanding.
monograph suggests a unified approach for talk-
— vi —
7. Chapter 1
Introduction �����������������������������������
n Review Methods ����������������������������������������
����������������������������������
n An Implementation Headset ������������������
���������������������
n Implementation Defined
n Paper Implementation
8. Chapter 1 • Introduction
Introduction
It has been well documented in many disciplines that major gaps exist between what is known as
effective practices (i.e., theory and science) and what is actually done (i.e., policy and practice).
Background & Purpose Han, & Weiss, 1995). Current views of imple-
mentation are based on the scholarly foundations
In the past few years several major reports prepared by Pressman & Wildavsky’s (1973) study
highlighted the gap between our knowledge of of policy implementation, Havelock & Havelock’s
effective treatments and services currently being (1973) classic curriculum for training change
“The ideas embodied
received by consumers. These reports agree that we agents, and Rogers’ (1983; 1995) series of analyses
in innovative social know much about interventions that are effec- of factors influencing decisions to choose a given
programs are not self- tive but make little use of them to help achieve innovation. These foundations were tested and
executing.” important behavioral health outcomes for chil- further informed by the experience base generated
dren, families, and adults nationally. This theme is by pioneering attempts to implement Fairweather
—Petersilia, 1990 repeated in reports by the Surgeon General (United Lodges (Fairweather, Sanders, & Tornatzky,
States Department of Health and Human Services, 1974) and National Follow-Through education
1999; 2001), the National Institute of Mental models (Stivers & Ramp, 1984; Walker, Hops, &
Health [NIMH] National Advisory Mental Health Greenwood, 1984), among others. Petersilia (1990)
Council Workgroup on Child and Adolescent concluded that, “The ideas embodied in innovative
Mental Health Intervention Development and social programs are not self-executing.” Instead,
Deployment (2001), Bernfeld, Farrington, & what is needed is an “implementation perspective
Leschied (2001), Institute of Medicine (2001), on innovation—an approach that views postadop-
and the President's New Freedom Commission tion events as crucial and focuses on the actions
on Mental Health (2003). The authors call for ap- of those who convert it into practice as the key to
plied research to better understand service delivery success or failure” (p. 129). Based on their years
processes and contextual factors to improve the of experience, Taylor, Nelson, & Adelman (1999)
efficiency and effectiveness of program implemen- stated, “Those who set out to change schools and
tation at local, state, and national levels. schooling are confronted with two enormous tasks.
Our understanding of how to develop and The first is to develop prototypes. The second
evaluate evidence-based intervention programs has involves large scale replication. One without the
been furthered by on-going efforts to research and other is insufficient. Yet considerably more atten-
refine programs and practices, to define “evidence tion is paid to developing and validating prototypes
bases” (e.g., Burns, 2000; Chambless & Ollendick, than to delineating and testing scale-up processes.
2001; Lonigan, Elbert, & Johnson, 1998; Odom, Clearly, it is time to correct this deficiency.” (p.
et al., 2003), and to designate and catalogue 322). Gendreau, Goggin, & Smith (1999) added
“evidence-based programs or practices” (e.g., the that, “we cannot afford to continue dealing with
National Registry of Evidence-based Practices and the business of program implementation and
Programs, Substance Abuse and Mental Health related technology transfer topics in a cavalier
Services Administration, n.d.; Colorado Blueprints fashion” (p. 185).
for Violence Prevention, Mihalic, Fagan, Irwin, The purpose of this monograph is to describe
Ballard, & Elliott, 2004). However, the factors the results of a far-reaching review of the imple-
involved in successful implementation of these mentation literature. There is broad agreement that
programs are not as well understood (Backer, 1992; implementation is a decidedly complex endeavor,
Chase, 1979; Leonard-Barton & Kraus, 1985; more complex than the policies, programs, pro-
Reppucci & Saunders, 1974; Rogers, 1983, 1995; cedures, techniques, or technologies that are the
Shadish, 1984; Stolz, 1981; Weisz, Donenberg, subject of the implementation efforts. Every aspect
—2—
9. Implementation Research: A Synthesis of the Literature
of implementation is fraught with difficulty, from Literature with any data (quantitative or
system transformation to changing service provider qualitative) and any design (surveys to high qual-
behavior and restructuring organizational contexts. ity randomized group designs or within subject
Given the importance of implementation, the pur- designs) in any domain (including agriculture,
pose of this review is to create a topographical map business, child welfare, engineering, health,
of implementation as seen through evaluations of juvenile justice, manufacturing, medicine, mental Given the state of the
factors related to implementation attempts. It is not health, nursing, and social services) was eligible field, the goal was
an attempt to be exhaustive. Some literature reviews for inclusion. to “review loosely”
have very exacting criteria and review procedures, a Databases searched included PsycINFO,
to capture meaning,
style well-suited to areas of well-developed knowl- Medline, Sociological Abstracts, CINAHL,
edge. With respect to implementation, there is no Emerald, JSTOR, Project Muse, Current detect relationships
agreed-upon set of terms, there are few organized Contents, and Web of Science. Once the research among components,
approaches to executing and evaluating implemen- team had completed the literature search, nearly and help further the
tation practices and outcomes, and good research 2,000 citations were retrieved and entered into development of the
designs are difficult when there are “too many an EndNote database. The principal investigators
practice and science
variables and too few cases” (Goggin, 1986). Given then proceeded to pare down the list by reading
the state of the field, the goal was to “review loosely” the titles and abstracts using the same guidelines of implementation.
to capture meaning, detect relationships among for citation retrieval (full details are provided
components, and help further the development of in Appendix A). The remaining citations (N =
the practice and science of implementation. 1,054) were retrieved for full-text review and
The remainder of this introduction sets the content analysis. The review team developed a
stage for reading the monograph. There is an over- data extraction tool called the article summary to
view of the review methods in order to provide the record pertinent information from each docu-
reader with a context for evaluating the face validity ment reviewed. The article summary covered
of the review in terms of scope, findings, and several aspects including: the research domain,
frameworks. This is followed by an orientation to topic or purpose of the article, methods, results
implementation as distinct from program develop- and findings, codes or stages of implementation
ment and a definition of implementation. as defined by the codebook, selected quotations,
selected references, and memos or notes made by
Review Methods the reviewer about the article.
Full text reviews were completed by one of
The goal of this literature review is to syn- the five review team members. Each team member
thesize research in the area of implementation as was asked to make note of any particularly note-
well as to determine what is known about relevant worthy or “significant” implementation articles in
components and conditions of implementation. the memo section of the article summary if it met
Search strategies were developed by the research one of the following three criteria: (1) well-de-
team as an iterative process in consultation with signed experimental evaluations of implementa-
the Louis de la Parte Florida Mental Health tion factors, (2) careful reviews the implementa-
Institute (FMHI) University of South Florida tion literature, or (3) well-thought-out but more
librarian. The research team began the literature theoretical discussions of implementation factors.
searching process by establishing guidelines for For example, “significant” articles included
citation retrieval. The following citation retrieval literature describing group or within-subject
criteria were used to select reports, books, and experimental designs, meta-analyses, or literature
published and unpublished article citations for reviews pertaining to specific implementation
preliminary review: factors; literature describing useful frameworks
• published in English no earlier than 1970, or theoretical summaries; or qualitative analyses
• the title or abstract contained one or more of of specific implementation efforts. Literature that
the search terms, and focused on author-generated surveys of those
involved in implementation efforts, focused on
• an empirical study, meta-analysis, or literature interventions and only provided incomplete de-
review.
—3—
10. Chapter 1 • Introduction
scriptions of implementation factors, or primarily An Implementation Headset
presented the opinions of the authors were not
included as “significant” articles. It is important to have an “implementation
After reading the full text, about 30% of the headset” while reading this monograph. From an
1,054 articles were dropped from the review. Most implementation point of view, there are always
often, deletions occurred when implementation two important aspects of every research study,
was mentioned in the title or abstract but was demonstration project, or attempted intervention.
not evaluated in any way in the article itself (i.e., In each study, there are intervention processes and
The lack of common was not “an empirical study, meta-analysis, or outcomes and there are implementation processes
definitions and review”). Once the full text review was completed, and outcomes. When implementing evidence-
the lack of journals 743 articles remained, about half (377) of which based practices and programs, Blase, Fixsen, &
were identified as significant implementation Phillips (1984) discussed the need to discriminate
specifically oriented implementation outcomes (Are they doing the
articles. Of these, 22 articles reported the results
to implementation of experimental analyses (randomized group program as intended?) from effectiveness outcomes
research probably reflect or within subject designs) or meta-analyses of (Yes, they are, and it is/is not resulting in good
the poorly developed implementation variables. Article summaries outcomes.). Only when effective practices and
state of the field. were sorted into content areas by searching across programs are fully implemented should we expect
articles for the codes described in the codebook positive outcomes (Bernfeld, 2001; Fixsen & Blase,
(see Appendix B). The principal investigators then 1993; Institute of Medicine, 2001; Washington
proceeded to review each area for common imple- State Institute for Public Policy, 2002).
mentation themes and patterns. So far, as the wave of interest in evidence-
The review was challenging due to the lack of based practices and programs has swept across
well-defined terms. Diffusion, dissemination, and human services, the nature of the evidence about
implementation sometimes referred to the same interventions has received the preponderance of
general constructs and, at other times, quite dif- attention from researchers and policy makers. As
ferent meanings were ascribed to the same terms. Kitson, Harvey, & McCormack (1998) stated, “...
For example, “implementation” sometimes means the investment in developing structures to ensure
“used” in a general sense or “put into effect” gold standard research evidence has yet to be
with specific reference to a program or practice. matched by equal investment in ways of elucidat-
At other times it referred to a set of methods to ing how organizations change cultures or use dif-
purposefully help others make use of a program ferent techniques to manage the change process”
or practice on a broad scale. Similarly, coaching, (p 157). From an implementation point of view,
supervision, academic detailing, and on-the-job doing more and better research on a program or
teaching were used to describe similar activities. practice itself does not lead to more successful
Are the “implementers” the ones teaching or the implementation. A series of meta-analyses and
ones being taught? The answer is, it depends on detailed assessments of the strength of research
the author. We have created our own lexicon with findings for certain practices and programs may
definitions (see Appendix A and B) in the text to help a consumer, agency, or community select
help guide the reader through this monograph a program. However, more data on program
and to reduce confusion. The lack of common outcomes will not help implement that program.
definitions and the lack of journals specifically ori- Implementation is an entirely different enterprise.
ented to implementation research probably reflect Thus, an intervention must be well defined and
the poorly developed state of the field. carefully evaluated with regard to its effects on its
intended consumers (children, families, adults).
Likewise, implementation of an intervention
must be well defined and carefully evaluated with
regard to its effects on its intended consumers
(practitioners, managers, organizations, systems).
An implementation headset also is critical for
understanding and interpreting data from outcome
—4—
11. Implementation Research: A Synthesis of the Literature
studies. Rossi & Freeman (1985) identified three the 6% found by Rogers-Weise in 88 group-de-
ways in which inadequate measures of program sign parent training studies published from 1975
implementation may lead to an incorrect con- to 1990, and the 14.9% noted by Gresham et
clusion that an intervention is ineffective. First, al. (1993) in evaluations of behaviorally based For the purposes of
no treatment or too little treatment is provided; interventions published from 1980 to 1990” (p. this review:
second the wrong treatment is provided; and third, 41). Dane & Schneider (1998) concluded that, “A Implementation is
the treatment is nonstandard, uncontrolled, or var- reorganization of research priorities is needed to
ies across the target population. Dobson & Cook facilitate less confounded, better quality evalua-
defined as a specified
(1980) described “type III” (type three) errors. That tions of preventive interventions” (p. 42). set of activities designed
is, evaluating a program that was described but not Thus, implementation variables are not to put into practice an
implemented. In their analysis of a program for synonymous with those involved in interventions activity or program of
ex-offenders, they found only 1 in 20 consumers and implementation outcomes are important known dimensions.
actually received the program as described in the to measure, analyze, and report when attempt-
methods section. Thus, the outcome data could ing to interpret research findings or broad scale
not be attributed to the program as described. applications (Bernfeld, 2001; Blase et al., 1984;
Feldman, Caplinger, & Wodarski (1983) found Dusenbury, Brannigan, Falco, & Hansen, 2003;
that apparent findings of no differences among Forsetlund, Talseth, Bradley, Nordheim, &
groups were explained by measuring the applica- Bjorndal, 2003; Goodman, 2000; Mowbray,
tion of the independent variables. Those youths Holter, Teague, & Bybee, 2003; Rychetnik,
who were in groups whose leaders skillfully fol- Frommer, Hawe, & Shiell, 2002).
lowed the protocol had better outcomes.
Outcome interpretation is further compro- Implementation Defined
mised when control groups utilize the compo-
nents of the evidence-based program or practice, What is “implementation?” For the purposes
or, if the experimental programs fail to implement of this review, implementation is defined as a
key aspects of the intervention. In studies of one specified set of activities designed to put into
evidence-based program (Assertive Community practice an activity or program of known dimen-
Treatment or ACT; Bond, Evans, Salyers, sions. According to this definition, implementa-
Williams, & Kim, 2000) it was found in one tion processes are purposeful and are described
case that a control site had incorporated many in sufficient detail such that independent observ-
ACT principles (McHugo, Drake, Teague, & Xie, ers can detect the presence and strength of the
1999), while in another that the experimental sites “specific set of activities” related to implementa-
had implemented fewer aspects of the ACT model tion. In addition, the activity or program being
than expected (Bond, Miller, Krumweid, & Ward, implemented is described in sufficient detail so
1988). Dane & Schneider (1998) conducted a lit- that independent observers can detect its presence
erature review of prevention programs published and strength. As noted earlier, when thinking
between 1980 and 1994. They found that only about implementation the observer must be aware
39 (24%) of 162 outcome studies documented of two sets of activities (intervention-level activity
the implementation of the independent variables and implementation-level activity) and two sets of
(i.e., fidelity) and only 13 used a measure of fidel- outcomes (intervention outcomes and implemen-
ity as a variable when analyzing the results. They tation outcomes).
also noted that the amount of documentation of The view becomes a bit more complicated
fidelity found in their review (24%), “compared when implementation-savvy researchers talk about
to the 20% found by Peterson, et al. (1982) in implementation-related “interventions” with
539 experimental studies published from 1968 to community leaders, agency directors, supervisors,
1980 in the Journal of Applied Behavior Analysis, practitioners, policy makers, and funders. For pur-
the 18.1% found by Moncher and Prinz (1991) poses of this monograph, we will use “interven-
in 359 treatment outcome studies published in tions” to mean treatment or prevention efforts at
clinical psychology, psychiatry, behavior therapy, the consumer level and “implementation” to mean
and family therapy journals from 1980 to 1988, efforts to incorporate a program or practice at the
—5—
12. Chapter 1 • Introduction
community, agency, or practitioner levels. Also, it implementation of that program or practice (e.g.,
is common to read about “implementation” of a funding, policy mandate). When faced with
program or practice as if it were an accomplished the realities of human services, implementation
fact when the context of the statement makes outcomes should not be assumed any more than
it clear that some process (more or less clearly intervention outcomes are assumed.
described) had been put in place to attempt the
Degrees of Implementation
During the course of the review, it was noted that various authors discussed the purposes
and outcomes of implementation attempts in different ways (Goggin, 1986). The purposes and
outcomes of implementation might be categorized as:
Paper implementation means putting into place new policies and procedures (the “recorded
theory of change,” Hernandez & Hodges, 2003) with the adoption of an innovation as the
rationale for the policies and procedures. One estimate was that 80-90% of the people-depen-
dent innovations in business stop at paper implementation (Rogers, 2002). Westphal, Gulati,
& Shortell (1997) found in their survey of businesses that, “If organizations can minimize
evaluation and inspection of their internal operations by external constituents through adop-
tion alone, they may neglect implementation altogether, decoupling operational routines from
formally adopted programs.” (p. 371). Thus, paper implementation may be especially preva-
lent when outside groups are monitoring compliance (e.g., for accreditation) and much of the
monitoring focuses on the paper trail. It is clear that paperwork in file cabinets plus manuals
on shelves do not equal putting innovations into practice with benefits to consumers.
Process implementation means putting new operating procedures in place to conduct train-
ing workshops, provide supervision, change information reporting forms, and so on (the
“expressed theory of change” and “active theory of change,” Hernandez & Hodges, 2003)
with the adoption of an innovation as the rationale for the procedures. The activities related
to an innovation are occurring, events are being counted, and innovation-related languages
are adopted.
However, not much of what goes on is necessarily functionally related to the new prac-
tice. Training might consist of merely didactic orientation to the new practice or program,
supervision might be unrelated to and uninformed by what was taught in training, informa-
tion might be collected and stored without affecting decision making, and the terms used in
the new language may be devoid of operational meaning and impact. In business, this form
of implementation has been called the Fallacy of Programmatic Change. That is, the belief
that promulgating organizational mission statements, “corporate culture” programs, training
courses, or quality circles will transform organizations and that employee behavior is changed
simply by altering a company’s formal structure and systems (Beer, Eisenstat, & Spector,
1990). It is clear that the trappings of evidence-based practices and programs plus lip service
do not equal putting innovations into practice with benefits to consumers.
Performance implementation means putting procedures and processes in place in such a way
that the identified functional components of change are used with good effect for consumers
(the “integrated theory of change,” Hernandez & Hodges, 2003; Paine, Bellamy, & Wilcox,
1984). It appears that implementation that produces actual benefits to consumers, organiza-
tions, and systems requires more careful and thoughtful efforts as described by the authors
reviewed in this monograph.
—6—
13. Chapter 2
Implementation “The community both defines the
in the Context of problem to be solved and tests
the adequacy of the answer”
Community — Felner, 1997
n Research on Community Context
n Measuring Readiness
n Stages of Community Readiness
14. Chapter 2 • Implementation in the Context of Community
Implementation in the Context of
Community
Before we begin to delve into the mysteries of implementation, we want to affirm the obvious.
The World Bank advised Implementation occurs in the context of community.
that, “…for a mutually
reinforcing coalition to For present purposes, a “community” might Research on Community Context
emerge, each potential be members of a city, neighborhood, organization,
partner must make an service agency, business, or professional association. While those engaged in implementing pro-
A theme running throughout the literature was grams and practices consistently discuss the need
investment with a high the importance of knowing the current strengths for community readiness and buy-in, there are
degree of uncertainty and needs of a community prior to selecting and virtually no data to support any given approach
regarding the attempting to implement an innovation. In the to achieving buy-in. In addition, there are few
commitment, capacity, process of examining the community’s strengths studies that relate community preparation to later
and intentions of their and needs, a planning group often forms and be- implementation success. With respect to the con-
comes a catalyst for increasing awareness, mobiliz- cept of buy-in, several surveys of implementation
potential partner.” ing interests and driving planning activities. efforts in business and industry consistently found
The literature across domains consistently support for worker and other staff participation
cites the importance of “stakeholder involve- in decisions to make changes (e.g., Ramarapu,
ment” and “buy in” throughout all stages of the Mehra, & Frolick, 1995; Salanova, Cifre, &
implementation process (“Nothing about us Martin, 2004; Small & Yasin, 2000). Additional
without us” seems to apply to all stakeholders support was found in a longitudinal comparison
when choosing and implementing evidence-based study of worker stress and implementation of new
practices and programs as well as other treatment manufacturing technology. Korunka, Weiss, &
interventions). As summarized in an example by Karetta (1993) found subjectively-experienced
Petersilia (1990), “Unless a community recognizes stress decreases significantly following implemen-
or accepts the premise that a change in corrections tation in companies in which there was greater
is needed, is affordable, and does not conflict inclusion of employees in the planning process.
with its sentiments regarding just punishment, Stress levels were unchanged in companies with
an innovative project has little hope of surviving, lower levels of employee participation. For
much less succeeding” (p. 144). Fox & Gershman changes in businesses that rely heavily on human
(2000) summarized several years of experience interaction, Rogers (2002) emphasized the need
with the World Bank in its attempts internation- for communication, a clear theory of change that
ally to implement new policies to help the poor. makes the case for the intended changes in the
They advised that, “…for a mutually reinforc- organization; and the development of champions
ing coalition to emerge, each potential partner who can consistently advocate, cajole, recognize,
must make an investment with a high degree of reward, and encourage. Thus, buy-in supported
uncertainty regarding the commitment, capacity, by communication and internal champions was
and intentions of their potential partner” (p. 188). thought to be important by those involved in
many implementation processes and some evi-
dence points to benefits to those whose jobs were
changed in the process.
Working with communities and agencies
in preparation for implementing evidence-based
programs and practices also is seen as important
in human services (e.g., Adelman & Taylor, 2003;
Arthur & Blitz, 2000; Barber, Barber, & Clark,
—8—
15. Implementation Research: A Synthesis of the Literature
1983; Bierman et al., 2002; Cleaver & Walker, • Programs are perceived by teachers as practical,
2004; Crosby, 1991; Dennis, Perl, Huebner, & useful, and beneficial to students.
McLellan, 2000; Klem, 2000; Taylor et al., 1999). • Administrative support and leadership;
For example, Adelman & Taylor (2003) described instructional practice is valued by the school
some early stages of preparation for adopting in- leaders; administration provides long-term
novations in an educational setting: support for professional development of teach-
• Develop an understanding of the local big- ers and assessments of implementation and
picture context for all relevant interventions; student performance.
develop an understanding of the current status
Thus, mobilizing support and local champi-
of efforts; delineate how the innovation can
ons, community participation in decision making,
contribute with respect to the larger agenda;
developing understanding and commitment to an
articulate cost-effective strategies.
innovation, and clarifying feasibility and func-
• Mobilize interest, consensus, and support tions seem to be a few of the important aspects of
among key stakeholders; identify champions initiating implementation in a community.
and other individuals who are committed to
the innovation; plan and implement a social
marketing strategy to mobilize a critical mass Measuring Readiness
of support; plan and implement strategies to Some researchers are developing scales to
obtain support of key policymakers. measure “readiness” of practitioners. For example,
• Clarify feasibility; clarify how the functions Aarons (2004) has developed the “Evidence-based
can be institutionalized through existing, Practice Attitude Scale” to measure mental health
modified, or new infrastructure and operation- provider attitudes toward adopting evidence-based
al mechanisms; clarify how necessary changes practices and programs. The 18-item scale was
can be accomplished; formulate a long-range developed from the literature, consultation with
strategic plan. providers, and researchers with experience imple-
Similar community planning was deemed to menting evidence-based practices and programs.
be important to sustainability of innovations as The items assess the appeal of evidence-based
well. Denton, Vaughn, & Fletcher (2003) exam- programs, requirements for using evidence-based
ined a number of reading programs that had been practices and programs, openness to innovation,
widely implemented and identified the following and perceived divergence of evidence-based prac-
factors that seem to influence sustainability of tices and programs from usual practice. Clinical
high-quality implementation: and case management service providers from 51
• Teachers’ acceptance and commitment to the programs were surveyed and the results demon-
program; the presence of a strong school site strated good internal consistency and reliability.
facilitator to support them as the teachers Scales to measure organizational readiness
acquired proficiency in its execution. also are being developed (Lehman, Greener, &
Simpson, 2002; Simpson, 2002). Items on the
• “Unambiguous buy-in on the part of all staff at
Organizational Readiness to Change scale ask
the school” (p. 16); empower teachers to take
questions about motivational readiness (need for
ownership and responsibility for the process of
improvement, training needs, pressure to change),
school change; schools or districts must agree to
institutional resources (space, staffing, training,
follow procedures designed to ensure high-fidel-
computers, e-communication), staff attributes
ity implementation and agree to collect data on
(growth, efficacy, influence, adaptability), and or-
implementation and student outcomes
ganizational climate (clarity of mission and goals,
• Feelings of professionalism and self-determi- cohesiveness, autonomy, openness to communica-
nation among teachers; teachers are provided tion, stress, openness to change). Data collected
with professional development (training, from treatment staff in over 100 organizations
in-class coaching, and prompt feedback) that support the construct validity of the scales.
leads to proficiency.
—9—
16. Chapter 2 • Implementation in the Context of Community
A model for measuring readiness at the • Initiation: enough preparation has been done
community level also has been developed. Many to justify efforts, policies and actions are
Stages of Community of the readiness concepts found in the literature underway and still seen as new, enthusiasm is
Readiness were included in a Community Readiness Model high and problems (so far) are few.
developed by Edwards, Jumper-Thurman, Plested, • Stabilization: programs are up and running
Oetting, & Swanson (2000). In this model, assess- with support from administrators and com-
No Awareness
ment of the stage of readiness is done through key munity leaders, staff have been trained and are
informant interviews, with questions on six differ- experienced, limitations have been encoun-
ent dimensions related to a community’s readiness tered and resistance overcome.
Denial to mobilize to address a specific issue. Based on
The Community Readiness Model has been
experiences in working directly with communi-
used by researchers to help match communities in
ties, strategies for improving community readi-
preparation for experimental analyses of preven-
ness have been developed for each stage. Teams
Vague Awareness tion programs (Edwards et al., 2000). However,
of community members can use the strategies as
no psychometric testing was reported.
a guide to develop specific, culturally appropriate
In summary, community obviously is impor-
efforts that use local resources to help the commu-
tant to implementation and researchers are begin-
Preplanning nity to more advanced levels of readiness. Edwards
ning the process of developing measures of com-
et al. (2000) identified several stages of commu-
munity involvement in planning and implement-
nity readiness (some actions recommended by
ing programs and practices. Advice from those
the authors to improve community readiness are
Preparation engaged in implementation efforts emphasize the
provided in parentheses):
need for members of a community to recognize its
• No awareness: not a problem, just the way it
assets and needs, select interventions and services,
is. (Actions: Raise awareness of the issue via
build support and buy in, retain a monitoring
Initiation one-on-one visits with community leaders and
function, and help to assure long-term sustainabil-
members, visits with existing and established
ity of useful services. “Readiness” to implement
small groups to inform them of the issue, and
new practices and programs has intuitive appeal
one-on-one phone calls to friends and poten-
Stabilization but there is scant research evidence to support the
tial supporters.)
idea of “readiness” at any level (practitioner, or-
• Denial: some recognition of the problem but ganization, community). While the developers of
it is confined to a small group, we are helpless the various scales have assessed the reliability and
anyway. construct validity of their measures of readiness,
• Vague awareness: some recognition, some no- so far there has been no assessment of predictive
tion of doing something, no clarity. validity. Thus, the relationship between measures
• Preplanning: clear recognition of a problem, of readiness and later implementation success is
something needs to be done, leaders emerge, unknown. However, future research should be
but no specifics yet. (Actions: Raise awareness aided by including measures of readiness. The
with concrete ideas to combat the problem next step is to conduct research to determine the
by introducing information about the issue ways in which aspects of community or organiza-
through presentations and media, visit- tional preparation are related to later implementa-
ing and developing support in the cause by tion success.
community leaders, reviewing existing efforts
in community (programs, activities, etc.) to
determine who benefits and what the degree of
success has been, and conducting local focus
groups to discuss issues and develop strategies).
• Preparation: active planning with a focus on
details, leadership is active, resources are being
assessed and expanded.
— 10 —
17. Chapter 3
A Conceptual View ���������������������������������������
of Implementation ����������������������������������������
������������������������������
n Conceptual Framework �������������������������
n Purveyors
n Stages of Implementation
Defined
n What We Know about
Implementation Stages
n Experimental Analyses of
Implementation Strategies
18. Chapter 3 • A Conceptual View of Implementation
A Conceptual View of Implementation
A persistent problem encountered throughout this review of the implementation evaluation literature is the
lack of a common language and the lack of a common framework for thinking about implementation.
Conceptual Framework
Based on the review of the literature and 4. a FEEDBACK mechanism (a regular flow of
ideas from computer programming (Milojicic, reliable information about performance of
Douglis, Paindaveine, Wheeler, & Zhou, 2000) individuals, teams, and organizations acted
and creativity fields (Altshuller, 1984), we arrived upon by relevant practitioners, managers, and
at a conceptual framework for implementation of purveyors),
well-defined programs and practices. As shown in
5. that operate within a sphere of INFLUENCE
Figure 1, in its simplest form implementation has
(social, economic, political, historical, and psy-
five essential components:
chosocial factors that impinge directly or indi-
1. a SOURCE (a “best example,” often a com-
rectly on people, organizations, or systems).
posite of the original practice or program that
was developed and evaluated and the best Implementation components and out-
features of attempted implementations of that comes exist quite independently of the quality
practice or program), of the program or practice being implemented.
Ineffective programs can be implemented well
2. a DESTINATION (the individual practitio-
(e.g., the DARE program, Elliott, 1997; Ennett,
ner and the organization that adopts, houses,
Tobler, Ringwalt, & Flewelling, 1994). Effective
supports, and funds the installation and ongo-
programs can be implemented poorly (Fixsen
ing use of an innovation),
& Blase, 1993; Fixsen, Blase, Timbers, & Wolf,
3. a COMMUNICATION LINK (an individual 2001). Neither one is desirable. Desirable out-
or group of individuals, named “purveyors” comes are achieved only when effective pro-
in this monograph, representing a program or grams are implemented well (Fixsen et al., 2001;
practice who actively work to implement the Leschied & Cunningham, 2002; Washington
defined practice or program with fidelity and State Institute for Public Policy, 2002).
good effect at an implementation site), and
The essential implementation outcomes are:
Figure 1
A Conceptual Framework for Implementation of Defined 1. changes in adult professional behavior (knowl-
edge and skills of practitioners and other
Practices and Programs
Figure 1. Implementation Framework key staff members within an organization or
system),
Influence 2. changes in organizational structures and
Destination
cultures, both formal and informal (values,
philosophies, ethics, policies, procedures,
Communication decision making), to routinely bring about
Link and support the changes in adult professional
behavior, and
Source 3. changes in relationships to consumers, stake-
Feedback holders (location and nature of engagement,
inclusion, satisfaction), and systems partners.
— 12 —
19. Implementation Research: A Synthesis of the Literature
For example, Toyota Production Systems Another example is Multisystemic Therapy
(TPS) is a just-in-time manufacturing system (i.e., (MST; Henggeler & Borduin, 1990; Henggeler,
no unnecessary inventory at each input and output Schoenwald, Borduin, Rowland, & Cunningham,
stage) requiring massive reorganization of produc- 1998), a treatment for serious antisocial behavior
tion units, visual control and communication by in youth that is delivered via a homebased model
workers with other workers, and specific arrange- of service delivery.
ments of plant operating and management struc- SOURCE: MST methods were developed and
tures to support production teams and a consistent evaluated with serious and chronic juvenile of-
flow of materials (Kasul & Motwani, 1997). fenders by Henggeler, Borduin, and colleagues
SOURCE: Toyota created, developed, and evaluated in Missouri and South Carolina and is a well-
TPS methods at their auto manufacturing plants known evidence-based program.
in Japan and began replicating their system in DESTINATION: MST works with service sys-
affiliated parts manufacturing and auto assembly tems to identify those organizations that have
plants around the world. met certain criteria on a site-assessment instru-
DESTINATION: The Toyota Supplier and Support ment (e.g., leadership willing to adopt the MST
Center (TSSC) works with those organizations framework, “fit” of MST with the intended
found worthy of the total commitment required target population, adequate referral and fund-
to make the necessary changes. ing mechanisms, consonance of leadership and
clinician perception of the nature of MST, team
COMMUNICATION LINK: TSSC provides con-
structure with specific MST supervision weekly,
sulting and implementation support free of charge
accountability for outcomes at the therapist, su-
(e.g., analyzes the client’s manufacturing capability;
pervisor, and organizational levels; organizational
prescribes the best implementation strategy with
structures that support the team, willingness to
adaptations of some features of the TPS based on
examine outcomes systematically).
local circumstances and values; directly observes
and analyzes workers on the line, supply chains, COMMUNICATION LINK: MST Services,
etc.; identifies the key aspects at an operations Inc. is the official purveyor of the MST pro-
level; helps the plant redesign the workspace to gram nationally (e.g., information sharing, site
emphasize and conserve human motion, improve assessment, staff training, staff consultation and
safety, eliminate waste, and improve efficiency). coaching, staff evaluation).
FEEDBACK mechanisms: Task assignments are FEEDBACK mechanisms: The MST Institute
detailed and focused and the TSSC staff spend has a web-based system for collecting adherence
about 1 week per month for about 3 years observ- data monthly at the practitioner and supervisory
ing performance, reviewing progress, answering levels and using those data to inform decision
questions, and assigning new tasks until full making and consultation at the therapist and or-
implementation is achieved. ganizational levels. Adherence data are collected
monthly for the life of each implementation of
INFLUENCE: Car makers operate in an environ-
the MST program.
ment where consumers want a wide variety of
individually tailored products. To remain competi- INFLUENCE: There is increasing demand for
tive, manufacturers have to develop low-volume, evidence-based services to children and youth that
high variety production strategies that call for flex- can operate within typical organizational con-
ibility in manufacturing done via more automa- straints, funding sources, and referral streams while
tion and integration of processes on the floor. maintaining high fidelity and good outcomes.
The result is Toyota can deliver a high quality As a result, MST Services, Inc. has estab-
car equipped to the customer’s specifications within lished many high-fidelity implementation sites
21 days of the order being placed at a local dealer- that benefit youths and families across the country
ship, 2 to 3 times faster than the industry standard. and internationally.
— 13 —
20. Chapter 3 • A Conceptual View of Implementation
This conception of the implementation test of the program. The report of the findings sim-
processes helps to focus attention on the “mov- ply noted that the originators of the program had
ing parts,” that is, those aspects that help to bring received funding to provide technical assistance to
national programs and practices into contact with the implementation sites. Given the uneven results,
practitioners who can provide direct benefit to it is unfortunate that there was no link back to
consumers locally. The generality of the con- purveyor activities. Nevertheless, in all of these in-
cepts presented in Figure 1 is highlighted by the stances, a purveyor works in more or less organized
A Purveyor is an examples from manufacturing and human services ways with the intention to implement a specified
individual or group of and applies with equal ease to a wide variety of practice or program at a particular location. Over
individuals representing programs and practices in agriculture, business, the years a purveyor also has been described as a
child welfare, engineering, health, juvenile justice, “change agent” (Fairweather et al., 1974; Havelock
a program or practice
manufacturing, medicine, mental health, nursing, & Havelock, 1973), “linking agent” (Kraft, Mezoff,
who actively work to and social services. The information in the fol- Sogolow, Neumann, & Thomas, 2000), “program
implement that practice lowing chapters is organized around the concepts consultant” (Gendreau et al., 1999), and “site coor-
or program with fidelity contained in Figure 1. dinator” (Blase et al., 1984).
and good effect. An advantage of having a well organized
Purveyors and persistent approach to implementation of
evidence-based practices and programs may be
As a communication link, in this monograph, that the purveyor can accumulate knowledge over
we make use of the notion of a “purveyor.” By that time (Fixsen & Blase, 1993; Fixsen, Phillips, &
we mean an individual or group of individuals rep- Wolf, 1978; Winter & Szulanski, 2001). Each
resenting a program or practice who actively work attempted implementation of the program reveals
to implement that practice or program with fidelity barriers that need to be overcome and their (even-
and good effect. Thus, in the examples above, the tual) solutions. Problems encountered later on
Toyota Supplier and Support Center (TSSC) is may be preventable with different actions earlier
a purveyor of the Toyota Production Systems for in the implementation process. Thus, with experi-
manufacturing automobiles. MST Services, Inc. is ence, the purveyor group can learn to change their
the purveyor of the Multisystemic Therapy (MST) approaches early in the process and avoid some
program for serious and chronic juvenile offenders. of the later problems. In addition, an experienced
These are clear-cut examples of purveyors and each purveyor can describe to the managers of an
has a set of activities designed to help new organi- implementation site the likely problems that will
zations (“implementation sites”) implement their arise and the likely solutions that can be applied.
respective programs. In other cases, the “purveyor” This seems to engender confidence and may lead
is not so readily identified nor are the activities well to greater persistence to “see it through” when the
described. For example, the Assertive Community going gets rough during the early stages of imple-
Treatment program and the Wraparound approach mentation. The problem is that the feedback loops
seem to have several individuals who act as con- for implementation efforts are very long. It often
sultants to communities and agencies interested in takes years to develop an implementation site and
adopting those programs. The Wraparound group then see how well that site performs with respect
has recognized the problem of multiple defini- to implementation outcomes and intervention
tions of their approach being used by different outcomes and a few more years to adjust strategies
purveyors and have formed a national association and experience new results in an ongoing itera-
to develop a common definition of the approach tive process (Blase et al., 1984; Fixsen & Blase,
and a common set of processes for assessing the 1993; Fixsen et al., 2001). Having a consistent
fidelity of new implementation sites (Bruns, Suter, group involved as purveyors of a given program or
Leverentz-Brady, & Burchard, 2004). The literature practice may provide a repository for (more or less
is not always clear about the activities of a purveyor. carefully evaluated) experiential knowledge and
For example, the Quantum Opportunity Program wisdom accumulated from a series of (more or less
(Maxfield, Schirm, & Rodriguez-Planas, 2003) was successful) implementation attempts over many
implemented in several sites in a major, multi-state years (Schofield, 2004).
— 14 —
21. Implementation Research: A Synthesis of the Literature
Stages of Implementation Defined Cohen, Farley, Bedimo-Etame, Scribner, Ward, Stages of the
Kendall, & Rice (1999) describe the use of similar Implementation Process
As implied in Figure 1, implementation is a strategies to increase the availability and use of
process, not an event. Implementation will not condoms in one state. The processes of mapping
happen all at once or proceed smoothly, at least consumer needs and understanding the enabling Exploration and Adoption
not at first. Based on their analyses of franchised and limiting aspects of the contexts in which
businesses, Winter & Szulanski (2001) stated that, interventions can occur seem to be important
“We treat knowledge transfer as a process (not a during the exploration process. At the end of the Program Installation
one-time act) by which [a purveyor] recreates a exploration stage, a decision is made to pro-
complex, causally ambiguous set of routines in ceed with implementation of an evidence-based
new settings and keeps it functioning. The [pur- program in a given community or state based on
veyor] gradually hones its ability to manage such formal and informal criteria developed by the Initial Implementation
a process through experience and repetition” (p. community and by the evidence-based program
741). Thus, a purveyor (COMMUNICATION (Blase et al., 1984; Khatri & Frieden, 2002;
LINK) can help organizations and systems stay Schoenwald & Hoagwood, 2001). The point of
on track and can help recognize and solve com-
Full Operation
entry for evidence-based practices and programs
mon implementation problems in a timely and may be at the system level or at the provider level.
effective manner. The following appear to be As discussed in Chapter 2, broad-based com-
discernible stages in the process of implementing munity education and ownership that cuts across Innovation
evidence-based practices and programs (e.g., Blase service sectors may be critical to installing and
& Fixsen, 2003; Cheung & Cheng, 1997; Faggin, maintaining an evidence-based program with its
1985; Feldman, Baler, & Penner, 1997; Fox & unique characteristics, requirements, and benefits.
Sustainability
Gershman, 2000; Rogers, 2002; Williams, 1975; Kraft et al., (2000) describe a “pre-implementa-
Zins & Illback, 1995). tion” stage for implementing HIV/AIDS preven-
tion programs where service providers, commu-
Exploration and Adoption nity planning groups, advisory boards, consumer
population members, related organizations, and
At some point, someone has to think about purveyors meet and exchange information to:
making use of an innovation. This requires some • identify the need for an intervention consider-
degree of awareness that leads to acquisition of ing the information available
information and exploration of options. A large
and varied literature exist describing “diffusion” • acquire information via interactions with one
of information and how individuals and orga- another
nizations make “adoption decisions” (Rogers, • assess the fit between the intervention program
1983; Westphal et al., 1997; Fitzgerald, Ferlie, & and community needs
Hawkins, 2003). Rogers’ work has been influ- • prepare the organization, staff, and resources
ential and often is cited as the conceptual model by mobilizing information and support.
used by others. The purpose of exploration is to
For the Multidimensional Treatment Foster
assess the potential match between community
Care Program (Chamberlain, 2003), the purveyor
needs, evidence-based practice and program
begins by assessing the readiness of the interested
needs, and community resources and to make a
agency with questions about the agency’s history,
decision to proceed (or not). Social marketing
current resources, current staffing patterns, and
methods seem to be relevant to the exploration
relationships with key stakeholders. In addition,
process. Social marketing emphasizes knowing
they assess potential barriers to implementation
consumer needs and matching interventions with
relating to funding, staffing, referrals, and foster
those needs (Andreasen, 1995). Flocks, Clarke,
parent recruitment. The result of the exploration
Albrecht, Bryant, Monaghan, & Baker (2001)
stage is a clear implementation plan with tasks
provide a detailed description of social market-
and time lines to facilitate the installation and
ing strategies applied to reducing the adverse
initial implementation of the program.
effects of pesticide exposure among farm workers.
— 15 —
