1. Chris Nicoletti
Activity #267: Analysing the socio-economic
impact of the Water Hibah on beneficiary
households and communities (Stage 1)
Impact Evaluation
Training Curriculum
Session 4
April 19, 2013

2. MEASURING RESULTS
From Promises into Evidence
IMPACT EVALUATION
AND
This material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledge its
use as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material,
The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarily
those of the World Bank.
Some of the data collection management material was developed by Adam Ross

3. 3
Tuesday - Session 1
INTRODUCTION AND OVERVIEW
1) Introduction
2) Why is evaluation valuable?
3) What makes a good evaluation?
4) How to implement an evaluation?
Wednesday - Session 2
EVALUATION DESIGN
5) Causal Inference
6) Choosing your IE method/design
7) Impact Evaluation Toolbox
Thursday - Session 3
SAMPLE DESIGN AND DATA COLLECTION
9) Sample Designs
10) Types of Error and Biases
11) Data Collection Plans
12) Data Collection Management
Friday - Session 4
INDICATORS & QUESTIONNAIRE DESIGN
1) Results chain/logic models
2) SMART indicators
3) Questionnaire Design
Outline: topics being covered

4. 4
Using a Results Chain
What are the intended results of the
program?
How will we achieve the intended
results?
How will we know we have achieved
the intended results?
A Results Chain answers 3 questions:
Impact Evaluation Training Curriculum - Activity 267

5. 5
The Results Chain in a Typical Program
o Budget
o Staffing
o Training
o Studies
o Construction
o Training plan
completed
o Cash transfer
delivered
o Road
constructed
o School built
o New practices
adopted
o Use of the road
o School attendance
up
o Health service use
up
o Poverty reduced
o Income inequality
reduced
o Labor productivity
increased
Results-based management
Implementation Results
Financial,
human, and
other resources
mobilized to
support
activities.
Actions taken or
work performed
to convert inputs
into specific
outputs.
Project
deliverables
within the control
of implementing
agency
SUPPLY SIDE.
Use of outputs by
beneficiaries and
stakeholders outside
the control of
implementing agency
DEMAND SIDE.
Changes in
outcomes that have
multiple drivers.
INPUTS ACTIVITIES OUTPUTS OUTCOMES
LONGER-TERM
OUTCOMES
HIGHER ORDER
GOALS

6. 6
Example 1: Results Chain
Activities Outputs Outcomes
Longer-term
Outcomes
Education
o Teacher
training
o Textbooks
developed
o Teachers
trained in new
methods
o Textbooks
delivered
o New methods
used
o Increased
completion
rates
o Increased test
scores
o Increased labor
productivity
Health
o Doctors
hired
o Birth
attendants
trained
o New doctors
practicing
o Attendants
applying
methods
o Increased use
of health
clinics for
deliveries
o Improved
maternal
mortality
Social
Protection
and Labor
o CCTs
delivered
o Targeting
system
o MIS
o CCTs delivered
to target
households in
accordance
with conditions
o Increased food
consumption
o Increased
child health
visits
o Decreased
poverty
o Lower child
mortality

7. 7
Identify the sequence of inputs, activities, outputs
and outcomes:
Example 2: Results Chain
1
2
3
4
5
6
Information is available for parents about the
importance of breast feeding.
Children in community healthier.
Fewer children are having diarrheal diseases.
Mothers breast feeding rather than using formula.
New funds available to implement a health project
to reduce child malnutrition rates.
Design information campaigns on the importance
of breast feeding

8. 8
Identify the sequence of inputs, activities, outputs
and outcomes:
Example 2: Results Chain
1
2
3
4
5
6
Information is available for parents about the
importance of breast feeding.
Children in community healthier.
Fewer children are having diarrheal diseases.
Mothers breast feeding rather than using formula.
New funds available to implement a health project
to reduce child malnutrition rates.
Design information campaigns on the importance
of breast feeding
Input
Activity
Output
Outcome
Outcome
Outcome

9. Additional Land
Access to
Irrigation
Anchor Farmer
Cost of Inputs
# of Crops Grown
Value of Crop Mix
Access to Markets
Access to Additional
Inputs
Irrigation and Farm-
management
training
Contract Farming
and market
opportunities
Efficiency of
Irrigation and
farming practices
Anchor Farmer
Employment
Household Labor
Employment
Farm Production
and Output
On-Farm Income
Off Farm IncomeFarm Gate Prices
Primary Inputs Secondary Inputs Intermediate Impacts Outcomes
Key
Solid Line indicates positive
relationship
Dashed Line indicates
negative relationship
Orange Line indicates that
the relationship assumes
significant anchor farmer
Involvement
Example #3: Irrigation Activity – Logic Model

10. 10

11. 11
• Take moment to look at the Logic models that are
in the M&E plan to identify the parts that are
testable. Would you please identify the ones that
are testable, using impact evaluation?
• Remember, these are not processes and or monitoring
efforts.
• You are interested in the causal impact of an IndII
program on ________.
Activity #2 : Review the logic models
from the M&E Plan
Impact Evaluation Training Curriculum - Activity 267

12. 12
SMART: Identifying
good indicators
Specific
easurable
ttributable
argeted
ealistic
M
A
R
T

13. 13
Specific
easurableM
Measure as closely as possible what you want to know.
Outcome: Children treated for malaria
Indicators:
1. Increased utilization of clinics
2. Increased use of malaria drugs
Be clear about how it will be measured – specific.
Indicators:
1. % of health centers without stocks of drugs x, y & z for
more than a week at a time
2. % of health centers with availability of drugs
Source: Kathouri and Kusek, 2006

14. 14
ealisticR
Data obtainable at reasonable cost, frequency and accuracy.
Indicators:
1. HIV prevalence among 15-24 year-old
pregnant women
2. HIV prevalence among the total population
Source: Kathouri and Kusek, 2006
ttributableA
Logically and closely linked to a program’s efforts.
Indicators:
1. Life expectancy
2. % of children fully immunized at 1 year

15. 15
argetedT
Specific to the program’s target group.
Indicators:
1. % increase in employment
2. % increase in employment of graduates of
technical training center X, in the first year
after completion of training
Source: Kathouri and Kusek, 2006

16. 16

17. 17
• Are the components of the logic model that are
testable using SMART indicators? How might they be
refined for the purposes of impact evaluation?
• Focus on the testable part of the logic model that you
identified earlier.
• Think about the actual complexities of how you would
accurately and reliably measure your outcome of interest.
• Be creative and think about your own life… !
Activity #2 : Come up with a set of
SMART indicators for your project
Impact Evaluation Training Curriculum - Activity 267

18. 18
• What do you want to know?
• Tailor your survey to capture outcome indicators of
interest  Those that are suitable for impact
evaluation.
• Take the time, before you start developing your
questionnaire, to list EVERY indicator you are
interested in measuring, and every theoretical
characteristic that could affect those indicators.
– This should come from your logic model!
• Know how you will do your analysis!
• Don’t figure out how to do your analysis after the
fact… start early and have a plan in place.
Before you begin working on
the questionnaire…

19. 19
• Use reliable and valid instruments.
• If you are going to have surgery on your knee do you
want the doctor to use this… or this…?
Use valid instruments…

20. 20
• Often times, we will use existing survey
instruments that we know worked well.
• We will modify them for the cultural context that we are
working in.
• Adapt the instrument to the local language.
• A good question is one where the respondents
have the necessary knowledge to answer
• A good question is one where the respondent is
willing to provide the “true” answer.
• This is can be difficult with health- or income-related
questions.
Building of reliable
instruments.

21. 21
• Ask about first hand experience
• Ask one question at a time.
• Bad: Are you physically able to do things like walk or
carry a full water bucket without difficulty?
• Better: Are you physically able to carry a full water
bucket without difficulty?
• Obtain specific information, but do it simply
(example: total expenditures)
• Bad: How much do you spend per month on water?
• Better: How much do you spend on a single container
of water? How many containers of water do you buy
per week?
Developing a survey…

22. 22
• Use open questions for frequencies of
undesirable behavior.
• Use familiar words  important for health
questions.
• For socially-undesirable questions ask about the
past, before asking about the present.
• 1) Ask “have you ever done ______?”
• 2) “Are you currently doing ______?”
• Put sensitive questions in between less sensitive
questions, so the respondent does not get
overwhelmed.
Sensitive questions…

23. 23
• There are 4 types of questions
• Numeric
• Pre-coded
– Read the response options to the respondent
• Field-coded
– Do not read the response options to the respondent
• Verbatim
– Completely open-ended
TYPES OF QUESTIONS YOU
CAN USE…

24. 24
TYPES OF QUESTIONS
Example: On average How many days per week does [NAME] collect water for
drinking, cooking or bathing currently?
|__|
Example 2: What water source do you use most often for drinking?
1. WATER VENDOR (CART, WATER TRUCK, ETC.)
2. REFILL KIOSK
3. PUBLIC/COMMUNITY TAP
Example: What is your family name?
_________________________________________
What is each type of question?
Numeric
Pre-Coded
Field-Coded
Verbatim
E1.2. Does the plot belong to your HH or has it been rented / borrowed?
1 = belongs to HH
2 = rented
3 = borrowed

25. 25
• Lowercase vs. Uppercase
QUESTION CONVENTIONS
What treatments did you receive?
SEPARATE MULTIPLE RESPONSES WITH COMMAS
SEE CODES
Example of Field-Coded question:
How many days per week would [NAME] collect water for drinking,
cooking or bathing on average in January 2010?
-88 = DO NOT KNOW
-99 = NO RESPONSE
Example of Pre-Coded question:
What symptoms did [NAME] experience?
1 Abdominal Pain
2 Frequent Evacuation
3 Watery Feces
4 Bloody Feces
5 Vomiting
-99 NO RESPONSE
READ THE
LOWERCASE
LETTERS
DO NOT
READ THE
UPPERCASE
LETTERS

26. 26
• Parentheses
• Square brackets
• Underlined words
QUESTION CONVENTIONS
Example:
How old was [NAME] on (his/her) last birthday?
Example:
Is [NAME] responsible for collecting water for drinking, cooking or
bathing from a source outside of the home currently?
Example:
How many days per week would [NAME] collect water for drinking,
cooking or bathing on average in January 2010?

27. 27
• We use skip instructions to filter out irrelevant questions
• Skip patterns must be followed correctly
• Asking questions in a table
SKIP INSTRUCTIONS
Member ID
BASIC DEMOGRAPHICS OF HOUSEHOLD MEMBERS
B3-A B3-B B3-C B3-D B3-E B3-F B3-G B3-H
Please tell me
the name of
each of the
members of this
household,
starting with
yourself.
After yourself,
please list oldest
to youngest.
What is
[HOUSEH
OLD
MEMBER
S]’s sex?
1 Male
2 Female
What is
[NAME]’s
relations
hip to the
head of
househol
d?
SEE
CODES
How old is
[NAME]?
Years
-99 = No
Response
(IF B3-D>13
years)
Was [NAME]
employed in
January 2010?
1=Yes
2=No  B3-G
-88 = Do not
know
-99 = No
Response
(IF B3-D >13
years)
What type of
employment
did [NAME]
Have in January
2010?
SEE CODES
(IF B3-D>11
years)
Is [NAME]
employed
currently?
1=Yes
2=No  B3-I
-88 = Do not
know
-99 = No
Response
(IF B3-D >11
years)
What type of
employment
does [NAME]
Have
currently?
SEE CODES
01 |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_|
02 |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_|
D7-A Does your household have a piped water connection from the PDAM?
1 YES
2 NO  SKIP TO D7-B
|_|

28. 28
• It also helps to be aware of other impact
evaluations going on in your sector. You can
learn from their results when writing your survey.
• At NORC we have people whose entire career is
writing reliable questionnaires.
• There are a lot of biases that come from poor
questionnaires.
• Instrument bias – when the instrument you are using
does not capture the information you intend to capture.
This takes a lot of practice…

29. TEST YOUR QUESTIONNAIRE!
It saves you from having dodgy data!

30. 30
• Evaluate your questions – including the coding
that you use.
• Focus groups
• Cognitive Testing
• Think-aloud interviews
• Evaluate your questionnaire and procedures –
especially the timing of the survey and data flows.
• Conduct full field pretest.
• Revise questionnaire and procedures
Questionnaire testing

31. 31

32. 32
• Using one of the SMART indicators that you identified
previously, I would like for us to come up with a series
of questions to capture the information necessary for
this indicator.
• Think about question format – categorical, continuous,
verbatim, etc.
• Write out the questions and make sure they capture the
information you Intend. Are they confusing? Are they
ambiguous?
• Is it a complex indicator? If so, how will you break it into
pieces so the respondent can answer more easily?
– Think about your own ability to answer these questions.
Activity #3 : Questionnaire
development
Impact Evaluation Training Curriculum - Activity 267

33. 33
Tuesday - Session 1
INTRODUCTION AND OVERVIEW
1) Introduction
2) Why is evaluation valuable?
3) What makes a good evaluation?
4) How to implement an evaluation?
Wednesday - Session 2
EVALUATION DESIGN
5) Causal Inference
6) Choosing your IE method/design
7) Impact Evaluation Toolbox
Thursday - Session 3
SAMPLE DESIGN AND DATA COLLECTION
9) Sample Designs
10) Types of Error and Biases
11) Data Collection Plans
12) Data Collection Management
Friday - Session 4
INDICATORS & QUESTIONNAIRE DESIGN
1) Results chain/logic models
2) SMART indicators
3) Questionnaire Design
Outline: topics being covered

34. Thank You!

35. MEASURING RESULTS
From Promises into Evidence
IMPACT EVALUATION
AND
This material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledge its
use as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material,
The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarily
those of the World Bank.
Some of the data collection management material was developed by Adam Ross