A presentation by Rachel Steinacher, Research Manager for IPA-Kenya (Innovations for Poverty Action), on IPA and RCTsThis was presented at the Commission for Science and Technology (COSTECH) in Dar es Salaam, Tanzania, on June 19, 2014, to an audience of researchers.
2. Outline
Introduction to Innovations for Poverty Action
What we do and why
Different Types of Evidence
Process evaluation
Qualitative data
Experimental evidence
Why research impact
Determining and defining impact
What can we use the results of RCTs for?
Example: Primary School Deworming
Conclusion
Opportunities at Innovations for Poverty Action
3. Innovations for Poverty
Action
Founded in 2002, we partner with academics
to design and evaluate potential solutions to
global poverty problems using randomized
evaluations.
We also work to mobilize and support these
decisionmakers to use this evidence / these
solutions to build better programs and policies
at scale.
In collaboration with over 250 leading
academics and implementing organizations,
we have evidence from over 175 completed
studies with over 225 in progress around the
4. Innovations for Poverty
Action
Our Vision
More Evidence, Less
Poverty
Our Mission
To discover and
promote effective
solutions to poverty
problems around the
world
5. Different Types of
Evidence
We’ve been referring to ‘evidence’ but what we
mean might not be clear as there are many
different types of evidence
Types of evidence include (but are not limited
to!)….
Process Evaluation Data (Monitoring and
Evaluation)
Qualitative Data
Experimental Evidence
6. Process Evaluation
Process evaluation analyzes the extent to which program
operations, implementation, and service delivery are being
implemented properly.
When process evaluation is ongoing it is called program
monitoring (as in Monitoring and Evaluation: M&E).
Process evaluations help us determine, for example:
Whether services and goals are properly aligned
Whether services are delivered as intended to the appropriate recipients
How well service delivery is organized
The effectiveness of program management
How efficiently program resources are used
Process evaluations are often used by managers as benchmarks to
measure success, for example: the distribution of chlorine tablets is
reaching 80% of the intended beneficiaries each week.
These benchmarks may be set by program managers and
sometimes by donors.
7. Qualitative Data
Qualitative data is data that is captured in a non-numerical
way.
Collection can happen in many different ways, including:
In-depth interviews
One-on-one interviews with a participant with the purpose of probing
the thoughts, ideas, knowledge, etc. of the person being interviewed.
These will often be recorded and transcribed for analysis later.
Focus groups
Same as an in-depth interview, but includes multiple participants
Direct observation (ethnography, etc.)
This differs from interviewing as the person collecting the data does
not actively try to ask questions to the participant about the topic
being studied. They simply watch and observe, with the goal of
recording and interpreting what they say, their behaviour and
attitudes. Sometimes this involves recording (video and/or audio) or
writing notes about what they observe for analysis later.
Etc…
8. Qualitative Data
Qualitative data is useful to answer questions like:
Does this intervention ‘make sense’ in this context?
Is this intervention socially, politically, religiously
acceptable?
How do people perceive / understand this intervention
and the problem(s) it is meant to solve?
Hypothesis generation: what should we be studying /
collecting data on to better understand the intervention or
topic we’re interested in studying? Are we asking the
right questions?
9. Experimental Evidence –
Why Research Impact?
Experimental evidence allows us to determine
impact. By impact, we mean changes / outcomes
that have happened because the intervention
happened.
This let’s us know what interventions…..
Are effective [How effective is it? In what context?
For what demographic(s)? At what cost?]
Have no impact (Why wasn’t it effective? Was the
program implemented properly?)
Are harmful (Even something that seems like
common sense can have devastating consequences;
see the ‘Campbell-Somerville Study’)
10. Why Research Impact –
Influencing Programs and
Policy
We can use this information to try to influence
the decisions being made by governments,
NGOs, and other policymakers / program
implementers to make evidence-based
decisions
11. Determining Impact
Different research methods yield
different types of evidence….
Some of this evidence
demonstrates the impact of a
program
Some evidence cannot be used to
demonstrate impact. in fact, it can
lead us to believing things that
aren’t true, even though the data
seems to support a certain
conclusion….
Correlation does not imply
causation
13. Determining Impact
Impact is captured by analyzing a comparison between:
1. the outcome some time after the program has been
introduced
2. the outcome at that same point in time had the
program not been introduced (the ”counterfactual”)
16. Constructing the
Counterfactual
Counterfactual is often constructed by
selecting a group not affected by the
program
Randomized:
Use random assignment of the program to
create a control group which mimics the
counterfactual.
Non-randomized:
Argue that a certain excluded group mimics the
counterfactual.
17. Counterfactual
The counterfactual represents the state of the
world that program participants would have
experienced in the absence of the program
(i.e. had they not participated in the program)
Problem: Counterfactual cannot be observed
Solution: We need to “mimic” or construct the
counterfactual
19. Demonstrating Impact through
RCTs
RCTs are by far the most powerful research
methodology by allowing us to determine the
impact of a program by minimizing the
likelihood that the treatment and control
groups are meaningfully different from each
other
21. An example from IPA…
What can we use RCT results
for?
22. Primary School
Deworming
One of Innovations for Poverty Action’s most
famous studies, that is currently be scaled out
across the Kenya
23. Primary School
Deworming
Principal Investigators: Michael Kremer and
Edward Miguel
Partners:
Investing in Children and their Societies (ICS)
Location: Western Kenya
Sample: 30,000+ primary school children, 6-
18 years old
Timeline: 1997-2001
24. Primary School
Deworming
Policy Issue
Intestinal worms infect more than one in four
people worldwide and are particularly prevalent
among school-aged children in developing
countries.
These intestinal worms are believed to have a
negative impact on education, hindering child
development as well as school attendance and
reducing income later in life.
25. Primary School
Deworming
Context of the Evaluation
Busia district is a poor and densely-settled
farming region in western Kenya adjacent to Lake
Victoria. It has some of the country’s highest
worm infection rates.
One quarter of Kenyan student absenteeism is
attributed to abdominal pains which likely due to
intestinal worm infections. In addition, older
children may miss school to take care of siblings
who are sick with worm infections
26. Primary School
Deworming
Details of the Intervention
This study evaluated the Primary School Deworming
Project (PSDP), which was carried out by
International Child Support in cooperation with the
Busia District Ministry of Health.
The program randomly divided 75 schools into three
equal groups which were phased into treatment over
three years.
Within each group, a baseline parasitological survey
was administered to a random sample of pupils.
Schools with worm prevalence over 50% were mass
treated with deworming drugs every six months.
27. Primary School
Deworming
Details of the Intervention
In addition to medicine, treatment schools
received regular public health lectures, wall charts
on worm prevention, and training for one
designated teacher.
The lectures and teacher training provided information
on worm prevention behaviors—including washing
hands before meals, wearing shoes and not swimming
in fresh water.
28. Primary School
Deworming
Results and Policy Lessons:
Impact on Infection Intensity
Deworming reduced serious worm infections by
half amongst children in the treatment groups.
Pupils that received treatment reported being sick
significantly less often, had lower rates of severe
anemia, and showed substantial height gains,
averaging 0.5 centimeters.
29. Primary School
Deworming
Results and Policy Lessons:
Impact on School Attendance
Deworming increased school participation by at
least 7 percentage points, which equates to a
one-quarter reduction in school absenteeism.
When younger children were dewormed, they
attended school 15 more days per year, while
older children attended approximately 10 more
school days per year.
30. Primary School
Deworming
Results and Policy Lessons:
Treatment Spillover
The entire community and those living up to 6
kilometers away from treatment schools benefited
from “spillovers” of the deworming treatment.
Spillover effects occur because medical treatment
reduces the transmission of infections to other
community members.
Reductions in infection in non-treated children
resulted in an additional 3 to 4 days of schooling
per year.
31. Primary School
Deworming
Results and Policy Lessons
Including the spillover benefits of treatment, the
cost per additional year of school participation is
US$3.27, considerably less than the cost of many
alternative methods of increasing primary school
participation.
Creation of the ‘National School-based
Deworming Program’ and ‘Deworm the World’
34. Primary School
Deworming
Scaling-up the Program
Additionally, in 2007 the ‘Deworm the World
Initiative (DtW)’ was founded. It was managed
and operated through Innovations for Poverty
Action up until the end of 2013, when it our sister
organization, ‘Evidence Action’ spun off with the
goal of scaling evidence-based programs –
including DtW
To date, DtW’s programs have reached 37 million
children in 27 countries
Impact evaluation can and has had a profound
impact on the world
35. In Sum
There are many types of evidence that one
can collect and use to better understand an
intervention or topic
Experimental evidence generated by RCTs is
a powerful tool that we can use to promote
evidence-based development work by
influencing policymaking and program design.
36. Innovations for Poverty
Action Moving Forward
Innovations for Poverty
Action is a rapidly growing
organisation with plans to
expand in Tanzania (we
currently have 1 project
here)
We are always looking for:
Talented, innovative
researchers to partner with
on new research studies
For young researchers, this
typically means partnering
with a senior researcher to
allow for mentoring and
expert oversight
37. Innovations for Poverty
Action Moving Forward
We are also looking for bright, eager people
interested in working in working on our research
projects as:
Project Associates (entry-level managers with a
Bachelors degree + 0-2 years experience)
Project Managers (early to mid-career professionals
with a Bachelors degree + 2-5 years experience)
Research Managers (early to mid-career managers
with a Masters degree + 3-5 years of experience)
For more information, visit:
http://www.poverty-action.org/getinvolved/jobs