Implementation Research: A Primer

1. Implementation Research: A Primer
Stefan Baral, MD, MPH, CCFP, FRCPC (CM)

2. Overview
 Definitions
 Characteristics
 Conceptual Frameworks
 Evidence and Outcomes
 Methods
 Qualitative
 Quantitative
 Descriptive
 Analytic
 Experimental
 Conclusions
2

3. Definitions
 Implementation
 The use of strategies to introduce or change evidence-based health
interventions (policies, programs, individual practices) within specific
settings
 Implementation Science in HIV
 Implementation science is a multi-disciplinary field that seeks
generalizable knowledge about the behaviour of stakeholders,
organizations, communities, and individuals in order to understand the
magnitude, reasons for and strategies to close the gap between
evidence and routine practice for health in real world contexts
 Key Themes
 Multidisciplinary
 Generalizable
 Multiple stakeholders
 Closing gap between evidence and practice
 Real world contexts
3Lobb and Coldtiz, Implementation Science and Its Application to Population Health Annual Review of Public Health, 2013; Odeny, Padian, Doherty, Baral,
Beyrer, Ford, Geng, Definitions of implementation science used in the HIV/AIDS literature: a synthetic review. The Lancet Infectious Diseases, In Press, 2015

4. Implementation Research and Other PH Study Designs
4Source: Olakunle Alonge, Lobb and Coldtiz, Implementation Science and Its Application to Population Health Annual Review of Public Health, 2013;
https://catalyst.harvard.edu/pathfinder/t2detail.html

5. Characteristics of Implementation Research
5
 Findings are Warranted to Inform Policy/Program
 There is “sufficient evidence” to support the conclusions
of the work
 What is sufficient evidence?
 Transparency of Methods
 Support Critical Assessment of the Study
 Whether processes are adequate
 Conclusions justified
 Repeatability
 Don’t be afraid of “failure”
 A well done study is still a success in terms of
generating generalizable knowledge

6. Traditional Scientific Method
6http://www.sciencebuddies.org/science-fair-projects/project_scientific_method.shtml

7. Differences with IR
Competencies on a IR team:
 Research Methodologist
 Qual, Quant, Mixed Methods
 Ministry, Government, Agencies
 Either as members of team or study oversight
committee
 Health Professionals
 Involvement of health professionals from study settings
 Communications
 Public Health Professionals
 Health Commissioner/Associate Health Commissioner
 Public Health Inspector/Public Health Nurse
 Privacy Expert
 Stakeholder Assessment
 Community
7http://www.who.int/tdr/publications/year/2014/ir-toolkit-manual/en/

8. Traditional IR Approaches
8http://www.sciencebuddies.org/science-fair-projects/project_scientific_method.shtml
Active Consultation
Active Consultation
Appropriate Team,
Active Consultation
Active Consultation
Active Consultation
Active Consultation

9. IR Specific Objectives: Three Broad Areas
Three Broad Areas of IR Specific Objectives
1. Describe Health Situation or Interventions
2. Provide Data to Evaluate Ongoing
Interventions or Information Needed to Adjust
Interventions
3. Analyze missed targets and potential
solutions
9

10. Describe Health Situation and Intervention
 Magnitude of the problem
 Distribution of health needs of the population
 Risk factors for some problems
 People’s awareness of the problem
 Utilization patterns of services
 Cost-effectiveness of available and potential other
interventions
10

11. Evaluate Interventions
 Coverage of priority health needs
 Coverage of target groups
 Acceptability of the services
 Quality of services
 Cost-effectiveness of the intervention
 Impact of the program on health
11

12. Analyze Missed Targets
 Availability
 Acceptability
 Affordability
 Service delivery problems
 Fidelity
12

13. Evidence and Outcomes in
Implementation Research
Conceptual Frameworks
Outcomes
Evidence

14. Conceptual Frameworks Commonly used in IR
 RE-AIM
 Reach, Efficacy/Effectiveness, Adoption,
Implementation, Maintenance
 Stages of implementation
 National Implementation Research Network
 Exploration and Adoption, Program Installation (Prep),
Initial Implementation (pilot/adapt), Full
Implementation (>50% coverage), Sustainability
 Consolidated Framework for Implementation Research
 Intervention Characteristics, Inner Setting, Outer
setting, Individuals in the Intervention, Implementation
process
 Many others…. 14
Source: Glasgow et al 1999, National Implementation Research Network, 2005, Damschroder, 2009

15. Outcomes in Implementation Research
15
Clients
Outcome
Satisfaction
Symptomatology
Function
Population-
Based
Incidence of
diseases
Morbidity
Mortality
DALYs
Health
Outcomes
Efficiency
Coverage
Equity
Responsiveness
Services
Outcomes
Acceptability
Adoption
Appropriateness
Costs
Feasibility
Fidelity
Penetration
Sustainability
Implementation
Outcomes
Source: Olakunle Alonge, Proctor et al 2011

16. Implementation Outcomes
16
Implementation
Outcome
Working Definition* Related terms**
Acceptability Perception among stakeholders that an
intervention is agreeable
Related factors: (e.g. Comfort,
Relative advantage, Credibility)
Adoption Intention, initial decision, or action to try to
employ a new intervention
Uptake, Utilization, Intention to try,
Appropriateness Perceived fit or relevance of the intervention in a
particular setting or for a particular target
audience (e.g. provider or consumer) or issue
Relevance, Perceived fit,
Compatibility, Perceived usefulness or
suitability
Feasibility The extent to which an intervention can be
carried out in a particular setting or organization
Practicality, Actual fit, Utility, Trialability
Fidelity The degree to which an intervention was
implemented as it was designed in an original
protocol, plan, or policy
Adherence, Delivery as intended,
Integrity, Quality of programme
delivery, Intensity or dosage of
delivery
Implementation
cost
Incremental cost of the implementation strategy Marginal cost, Total cost***
Coverage Degree to which the population that is eligible to
benefit from an intervention actually receives it.
Reach, Access, Service Spread or
Effective Coverage, Penetration
Sustainability The extent to which an intervention is maintained
or institutionalized in a given setting.
Maintenance, Continuation,
Routinization Institutionalization,
Incorporation
Source: Proctor et al 2011; Peters, Adams, Alonge et al 2013

17. What is Sufficient Evidence?
 Evidence-based medicine is a global standard
 Double-Blinded (DB) RCT is gold standard
 Evidence-based PH interventions should also be a global
standard
 Often limited evidence, PH decision still needs to be
made
 Precautionary Principle for PH?
 When there are threats of serious or irreversible damage, lack of full
scientific certainty shall not be used as a reason for postponing
cost-effective measures to prevent environmental degradation
 To develop guidelines
 Need to characterize
 Efficacious
 Effective
 Sustainable and Scalable programs
17

18. Tension Between Internal and External Validity Challenges for Evidence
Combination Prevention
 Internal Validity
 Minimal study biases suggesting confidence in
ultimate conclusion of the study
 External Validity
 Generalizability of ultimate findings to broader
population
 Traditional Question for Clinicians/Programmers
 Does it work? What is effect size?
 Should I use it?
 Implementation Questions
 How, when, why, and where does it work?
 What factors influence effectiveness?
 Should I use it? How should I use it?

19. Traditional Research Pathway
 Effectiveness Research (and guideline development)
generally happens prior to implementation research
 Are there more time-effective approaches to integrate
implementation research with effectiveness/efficacy
research
 Assess barriers/facilitators to intervention uptake
 acceptance/adoption/routinization
 Diagnose quality gaps
 Fidelity
 Characterize Sustainability
 Maintenance, Cost-Effectiveness

20. Methods in Implementation Research

21. Research Objectives
 Qualitative Methods
 Exploration
 Using inductive methods to explore concepts, constructs,
phenomena, situations to develop hypotheses
 Quantitative Methods
 Explanation
 Characterizing the
 Relationship between concepts and phenomena
 Reasons for occurrences of events
 Prediction
 Use knowledge to forecast events
 Experiment
 Intervene/manipulate different settings or variables to produce
a desired outcome
 Mixed Methods
 Description
 Identify and describing the precursors/antecedents, nature, or
etiology of an outcome/phenomena

22. Qualitative Methods in Implementation Research
 To explore a health problem
 Evidence-based approach to characterizing individual, network, and
structural determinants of the health issue
 To identify variables that can later be measured
 Measuring the “right” implementation outcomes
 To develop a complex and detailed understanding of an issue
 Formative work to design the implementation plan for an effective
intervention
 Maximizing Acceptability, Appropriateness
 To understand the contexts or settings surrounding an
experience or phenomenon
 Case-Study of a program or a policy
 To understand the meaning behind an issue or experience
 Explore the “why” or “how” an intervention works or why not
 Fidelity

23. Quantitative Research Designs
 Observational
 Categories
 Descriptive
 Analytic
 Cross-Sectional
 Prospective
 Quasi-Experimental Studies
 no randomization of
individuals/communities/institutions
 Experimental
 Rapid Innovation in Implementation Design

24. Cohorts In Implementation Research
 Method
 Traditional “Active” Cohort includes
 Enrollment of participants and active follow-up of individuals
 Expensive to ensure retention
 “Passive” Cohorts (registries, EHR Repositories, etc)
 Rapidly becoming the standard in IR, but limitations include
quality and extent of data collection
 What is Different from Traditional Cohort Studies?
 Traditional Cohorts
 Incidence, Prevalence, Relationship between Exposure and
Outcome
 Cohorts in IR focus on measuring traditional IR Outcomes
 Acceptability, adoption, appropriateness, feasibility, fidelity
of interventions, implementation costs (cost-effectiveness),
Determinants of Coverage, Sustainability/Maintenance

25. Implementation Outcomes
Outcome of Interest Definition Data Source
Proportion of HIV-positive men who
enter care
At least one clinic visit attended
(where they see a clinician) after
positive test result within 3, 6 and
12-months of study visit
Clinic records, NHLS
documentation of lab tests
completed
Proportion of men who receive CD4
results
Receive POC CD4 results at study
visit or get CD4 tests at clinic and
return to receive results
Study visit CRFs, Clinic
records, SMS surveys
Proportion of ART-eligible men who
initiate ART
ART initiated by 12 months Study visit CRFs, Clinic
records
Time to ART initiation for ART-eligible
men
Length of time between receiving
positive HIV test result and CD4
<350 to initiating ART
Study visit CRFs, Clinic
records
Proportion of ART-eligible men who
initiate ART and are retained in care
 Attend 6-month clinic visit
(see a clinician)
 Attend 12-month clinic visit
(see a clinician)
 Attend 2+ clinic visits at least
3 months after within a 12-
month period
Clinic records, NHLS
documentation lab tests
completed (CD4, viral load,
others)
Proportion of treatment ineligible HIV-
infected men who receive a CD4 test
within 6 months following their study
visit CD4
Receipt of repeat CD4 test Clinic records, NHLS
documentation lab tests
completed (CD4, viral load,
others)

26. Implementation Outcomes
Outcome of Interest Data Source
Acceptability of outreach/CBO-based testing
intervention
Questions on study visit CRFs, surveys about why
participants chose to test
Relative advantages of non-clinic-based ART
initiation and retention package compared to
standard of care
CRFs, questions in surveys, qualitative data from both
participants and providers
Perceived credibility of CBOs to initiate ART as
compared to standard ART clinics
Survey indicators, qualitative data
Adoption of experimental interventions
including intention of use of decentralized
NIMART-trained nurse initiated ART and peer
navigator based support by participants
CRFs, survey indicators, qualitative data
Implementation costs associated with
experimental condition
Review of clinic-based budgets and ultimate costs to
assess marginal and total costs of interventions as
compared to standard of care
Maintenance and routinization of using clinic-
based approaches and peer-navigators for
retention as indicators to describe potential
sustainability of the interventions
Provider and participant qualitative data, survey
indicators

27. Experimental Studies
 Explanatory (Traditional Gold Standard)
 Understand and explain benefit of an intervention
under controlled conditions
 Maximize internal validity
 Pragmatic Trials
 Focus on the intervention in routine practice
 Intentional maximization of variability in how study is
implemented
 Variability of research settings (communities,
practice settings, types of providers, patients)
 Maximize external validity
 Adaptive Designs
 Emerging area of implementation research that
attempts to balance internal and external validity

28. Pragmatic Trials
 Testing a new intervention while maximizing external validity
 Formative Period
 Qualitative work, some descriptive or analytic observational work
 Consider different types of outcomes of effectiveness
 Directly Measured
 Health Outcomes
 Service Outcomes
 Implementation Outcomes
 Resources
 Institutional
 Human
 Financial
 Cost-effectiveness, Cost-Utility, Cost-
Minimization, Cost-Benefit, etc
 Indirect Assessment/Modeled Benefits
 Increasing use of Mathematical Models to Scale Results for
potential longer term outcomes, etc.

29. Effectiveness-Implementation Hybrid Trials
 Goal
 Measure markers of implementation and impact in the same
study
 Three Broad Designs
 Differentiated by prioritization of data collection
 Type 1
 1st priority - Impact of health intervention
 2nd Priority - gathering measures of implementation
 Feasibility/Acceptability using qualitative/mixed
methods
 Type 2
 Equal priority to impact and implementation
 Type 3
 1st priority – Implementation
 Fidelity of intervention, measures of adoption
 2nd Priority
 Impact of Health Intervention

30. Stepped Wedge Cluster Randomized Designs
 Method
 Assess baseline situation in all communities, but randomly phase
in intervention activities in steps, evaluating impact of intervention
time on outcomes
http://www.biomedcentral.com/1471-2288/6/54

31. Stepped Wedge Designs
 Pros
 Differences in exposure time allow each community site to
receive the intervention
 Ethics
 Mixed views on the ethics of stepped wedge
 Some believe more ethical to give intervention to all and
more feasible to implement, others believe trial not
warranted if success of intervention is certain and standard
of care can be justified so why not assess more cleanly
with parallel design
 Cons
 Analysis concerns around when an intervention starts – e.g. if a
community starts receiving the intervention today but takes 2
months to scale up and reach a substantial number of people,
exposure time will be diluted as everyone starts receiving the
exposure at the same time within the cluster
 If the outcome cannot be expected to happen over the time
period of one step, stepped wedge designs will be underpowered

32. Adaptive Designs For Implementation Studies
 Adaptive Intervention/Adaptive Implementation Strategy
 Specific decision rules for the implementation of an
interventions based on individual/community needs
 Trying to maximize both internal and external validity
 Trial Design
 Sequential, multiple assignment, randomized trials
(SMART)
 Use outcome data to inform the implementation of
the intervention being evaluated
 Can be at multiple steps
http://methodology.psu.edu/ra/adap-inter

33. SMART Study Example
Research Question: Among clinics not responding to Replicating
effective interventions/REP, how much does external or internal
facilitation help improve mood disorders program
Source: Kilbourne, Almirall, Implementation Science, 2014
 Adaptive Implementation Strategy
 A priori decisions about intervention based on response
 Randomize to control for confounders but being done in real world
setting
 Improved balance of Internal/External Validity
 Measure Implementation Outcomes Throughout

34. Power and Sample Size Calculations
 P&S Calculations for IS studies are complicated
 Powered to assess the “preponderance of evidence”
of the benefit of interventions
 Most realistic, but murky
 Powered for at least primary outcome (Eg. Viral
suppression)
 Cleaner, but is this really implementation
research?
 Powered on Outcome and Implementation Outcomes
 Limited resources, etc.

35. De-Implementation Science Studies
 The science of dissemination and implementation
confronts two problems
 Getting wider uptake of evidence-based interventions
in clinical or public health practice
 Elimination from clinical or public health practice of
tests and interventions that use resources without
enhancing patient outcomes
 As a field, we focus more on increasing interventions than
we do reducing unnecessary ones
 More incentive to discover new tools than to try and
more politically sensitive to try and remove services
for folks
 De-Implementation Science Methods
 Use many of the same experimental methods (CRCT,
SW, etc) but in reverse

36. Conclusions
 Implementation Research
 Seeks Generalizable Information intending to Close
Gap Between Evidence and Practice
 Tension Between Internal and External Validity
 What is most important for you and your stakeholders
 Rapid Evolution of Experimental Approaches in
Implementation Research
36