Shared at the 2017 American Evaluation Association Conference

1. Want to Integrate Gender in
your Evaluation but
Don’t Know Where to Start?
Jessica Fehringer, PhD, MHS
Brittany Iskarpatyoti, MPH
Mahua Mandal, PhD
MEASURE Evaluation, UNC-Chapel Hill
Sarah Frazer, MA
RTI International
November 10, 2017
American Evaluation Association

2. Key Definitions
SEX GENDER
Biological difference between
males & females:
• Universal for all human beings
• Unchanging
• Determined at birth
Beliefs about the appropriate roles,
duties, rights, responsibilities,
accepted behaviors, opportunities,
and status of women, men,
transgender people, others, in
relation to one another:
• Constructed by society
• Differs between cultures and over
time
• Acquired
• Includes identity, expression and
orientation
*Source: WHO 2009: Integrating gender into HIV/AIDS programmes in the health sector

3. Roles, relations, and power between men,
women, boys, girls, transgender people, and
others
• Focusing only on roles of women and girls ignores systems,
institutions, and structures in place, and the role of men and
boys within those systems
• Cannot fully empower women and girls without also
engaging men and boys
• Gender equality is good for everyone. Families and
communities benefit.
• Sexual and gender minorities exposed to same societal
expectations and can affect outcomes
Gender
Beyond Women and Girls

4. Equality vs. Equity
Equal treatment Fairness;
compensation

5. Why Integrate Gender into
Evaluation?
• Gender is everywhere
• Every development intervention has hierarchies,
judgments, etc. related to masculinity and femininity
• Provide evidence to:
• Improve programs
• Address gender dimensions of health, education,
agriculture, etc. (doesn’t have to be a “gender”
program)
• Raise awareness
• Identify gender related goals and priorities
based on available information and
consultation with stakeholders – better tailor
evaluation to needs and encourage data use

6. • Ensure all contribute to the research
management and process
• Ensure the research itself is gender sensitive
Why Integrate Gender into
Evaluation?

7. EnGendering Evaluation
“Not everything that counts can be counted,
and not everything that can be counted
counts.”
─Albert Einstein

8. 1. Identify stakeholders and their roles
2. Develop gender-integrated theory of
change
3. Define gender-related evaluation questions
4. Select appropriate study design
5. Select measures for gender-related outputs
and outcomes
6. Collect and interpret sex-disaggregated
and gender-sensitive data
7. Disseminate and use gender-integrated
results
7 Steps to EnGendering Evaluations

9. Step1. Identify Stakeholders
and Their Roles

10. Step1. Identify Stakeholders
and Their Roles
• Determine who the stakeholders are
• What is their role in the intervention?
• What do they gain?
• What might they lose?
• How much of a priority is their participation?
• At what stage should they be involved?
• What are their ways and capacities for
participation?
• Engagement strengthens
accountability, trust, credibility, data
use, and equity

11. Step 2. Develop a Gender-
integrated Theory of Change

12. Step 2: Theory of Change
Basic Structure Recap
If we do these
activities
Then we think
get this result
in the short
term
Then we think
we’ll get this
result in the
long term
Project Goal, based on solution creation, derived
from problem analysis
Short term outcome derived from thinking about the short
term steps that would lead to the long term impact
Activities derived from thinking about what concrete
actions will lead to the short term outcome

13. Step 2: Example Theories of Change
If we train
teachers
on
curriculum
delivery
Then
students
will receive
quality
instruction
on reading
Then
Students
age 7 to 10
will have
improved
literacy
Example Simple Education Project Theory of Change
If we train
teachers on
curriculum
delivery, with
attention to
gender-
related
differences in
learning
Then students
of all genders
will receive
quality
instruction on
reading
Then Students
of all gender,
age 7 to 10
will have
improved
literacy
Example Simple Gender-integrated Education Project Theory of Change

14. Step 2. Example Gender-
integrated Theory of Change
PROCESS OUTPUT OUTCOME IMPACT
Curriculum
for
discussions
with men on
basics of
importance
of FP
Discussion
series with
males about
importance of
family
planning
Number of
workshops
with men on
importance
of FP
Short-term:
Increased
awareness of
importance of
FP
Intermediate:
Increased
proportion of
men attending
FP clinics with
wives
Increased
proportion of
couples
using FP
INPUT

15. Step 3. Define Gender-
Related Evaluation Questions

16. Step 3. Define Questions
Consider:
• What is the evaluation mandate?
• How strongly gender-integrated is the
program?
• What barriers to changes in gender
norms is the program trying to address?
• How does gender play a role in theory
of change and what relationship(s)
might you want to examine?

17. Step 3. Define Questions
Outcome/impact:
 Has the program had the same impact on
health outcomes for men and women?
 Has the program reduced power differences
in relations between men and women?
 Have stigma and discrimination against
people who do not follow traditional gender
norms and behaviors been reduced?
 Has the removal of gender-based constraints
contributed to improved health outcomes?
Examples

18. Step 3. Define Questions
Process/implementation:
 Were gender-integrated program
components implemented as planned?
 What positive or negative unintended
effects on gender equality were identified
during implementation, if any? How were
they addressed?
 Were there any constraints (e.g., political,
practical, bureaucratic) to addressing
gender efficiently during implementation? If
yes, what efforts were made to overcome
these challenges?
Examples

19. Step 4. Select an
Appropriate Study Design

20. Step 4. Study Design
 Consider:
 What are your gender-related questions?
 Measure unintended consequences?
 Measure complex norms, relationships?
 How sensitive is topic/population?
 What types of evidence do stakeholders
need, $, time, other common
considerations
 Can your methods themselves be
empowering? (participatory methods)
Methods

21. Step 4. Study Design
 Quantitative
 What & aggregate picture
 Missing quality of
participation/performance
 Qualitative
 Why, how, quality
 Very important in gender
 Complex relationships hard to measure
quantitatively
 Inequality, inequity are lived
experiences
 Mixed methods ideal
Methods

22. To make determinations about effects of
program on men vs. women, boys vs. girls,
need large enough sample size of groups
of interest
Need to take into consideration in sample
size calculations, whether quantitative or
qualitative
Sampling
Step 4. Study Design

23. Step 4. Study Design
• Consider ethical concerns affecting
sampling; for example:
• GBV – only interview one woman per
household
• May need to use method like “snowball
sampling” for some stigmatized
populations
• E.g. -- Countries where homosexuality is illegal
and you are working with MSM
Sampling

24. Step 5. Select Measures for
Gender-Related Outputs and
Outcomes
Indicator types:
• Sex* and age-
disaggregated
• A minimum
• Gender-sensitive
* Ideally gender dis-aggregated

25. Gender-Sensitive Indicators
 No single “gold standard” for measuring
gender norms, attitudes, women’s
empowerment
 Use multiple measures
 Gender is a complex construct, and
operates in multiple
spheres
 A scale combining
several items
is more valid than a
single item used alone

26. Gender-Sensitive Indicators
 Household (HH) decision-making power
 Experience of gender-based violence
(GBV)
 Women’s autonomy and
empowerment
 Economic empowerment
 Service delivery: gender equity, stigma
and discrimination
 Couples counseling and male
involvement
Examples (1)

27. Gender-Sensitive Indicators
 Women’s Empowerment in Agriculture
Index (WEAI)- five domains (e.g. decisions
about agricultural production, time use,
and leadership in community)
 Perspectives on gender norms
 Gender Equitable Men (GEM) Scale;
example items:
 If someone insults a man, he should defend his
reputation with force if he has to”
 “A man needs other women even if things with his
wife are fine.”
Examples (2)

28. Gender-Sensitive Indicators
Demonstrate removal of gender-related barriers
• Barrier: perception that masculinity=many sexual
partners
• Measure success through:
• % of men/women agree men who have one sexual partner
are “real men”
• % of men who report that they have only one partners
Capture quality, not just quantity
• Not just training attendance but true participation
and decision-making roles
• % of trainees who have mastered relevant knowledge
• Not just #s employed but rather quality of jobs held
• % of management positions held by women
Selection tips

29. Step 6. Collect and Interpret
Sex-Disaggregated and
Gender-Sensitive Data
This Photo by Unknown Author is licensed under CC BY-NC-ND

30. Step 6. Collect
Consider:
• Logistics: Timing? Who is the interviewer?
Location?
• Questionnaire administration
• Gender biases that may affect data
• E.g. male over-reporting of # sexual partners,
underreporting of GBV
• Ethical issues, particularly for GBV and
other sensitive topics and marginalized
populations
• Review relevant international guidelines

31. Step 6. Interpret
 Go back to evaluation question(s)
before starting analysis
 Interpret within the context of literature
 Analyze data by sex and age,
minimum; including gender-sensitive
indicators (as applicable)
 If no differences, at least report that you
looked at subgroups and did not find
differences

32. Step 6. Interpret
 Identify patterns and potential issues
related to gender in program
implementation and outcomes
 Ask:
 If health outcomes are not occurring but
gender outcomes are, what might be
going on? And vice versa?
 Are gender elements of programs actually
happening?
 Don’t be seduced by personal beliefs
and expectations – conclude only what
results indicate

33. Step 7. Disseminate and Use
Gender-Integrated Results
MEASURE Evaluation. (2017) Gender Matters Video. Chapel Hill, NC: University of North
Carolina at Chapel Hill. See https://vimeo.com/219126971
0%
5%
10%
15%
20%
25%
Male Female

34. Step 7. Disseminate & Use
Gender-Integrated Results
Ensure that the information is available,
accessible, relevant, and useful
Also consider:
• Implications of sharing results
• E.g., sharing location-based results related to illegal
behavior
• How recommendations affect different
stakeholders positively or negatively

35. Group Exercise
1. Two case studies – one on HIV and
one on education
2. Divide into groups and choose a case
study
3. Review case study
4. Answer questions
5. Report back in 30 minutes

36. Gender Evaluation
Resources and Tools
 Compendium of Gender Equality and HIV
Indicators
 Violence Against Women and Girls
Compendium of M&E Indicators
 Trafficking in Persons and Health: A
Compendium of M&E Indicators
 Family Planning and Reproductive Health
Indicators Database (MEASURE Evaluation, PRH)
 HIV Indicator Registry (UNAIDS)
 Resource guide for gender data and statistics
(WHO, IGWG/USAID, & MEASURE Evaluation)
 Guidelines for Gender-Based Analysis of Health
Decision Making
 K4 Health IGWG Gender and Health Toolkit

37. Gender Evaluation
Resources and Tools
• Positive Youth Development Measurement
Toolkit
• Girl-Centered Program Design: A Toolkit to
Develop, Strengthen and Expand Adolescent
Girls Programs
• Integrating Gender in the Monitoring and
Evaluation of Health Programs: A Toolkit
• Integrating Human Rights and Gender Equality
in Evaluations (UN Evaluation Group)
• Gender Matters (video)(MEASURE Evaluation)
• Addressing Gender in Impact
Evaluation: What should be
considered? (ODI)

38. This presentation was produced with the support of the United States
Agency for International Development (USAID) under the terms of MEASURE
Evaluation cooperative agreement AID-OAA-L-14-00004. MEASURE
Evaluation is implemented by the Carolina Population Center, University of
North Carolina at Chapel Hill in partnership with ICF International; John
Snow, Inc.; Management Sciences for Health; Palladium; and Tulane
University. Views expressed are not necessarily those of USAID or the United
States government.
www.measureevaluation.org