1. Lend-A-Hand India: PLAN100
IMPACT ASSESSMENT PILOT STUDY:
METHODOLOGY AND PRELIMINARY
FINDINGS
BY IMRAN OOMER

2. Background
Lend-A-Hand India (LAHI) is developing a robust impact
measurement system to track the short-, medium-, and long term
impact on the students’ lives it touches through Plan100.
Current measurement focuses on New measurements will
quantitative measures of understand behavioral
student achievement: outcomes of its program, such as:
Student attendance • Increases in student confidence
Examination pass rates • Personal aspirations
Graduation rates • Entrepreneurial spirit

3. Project Objectives
 LAHI is at the initial stages of developing an impact
measurement system. This study will lay the groundwork
for it, and specifically accomplish the following:
• …a set of student outcomes expected from participation in
Define the IBT (Introduction to Basic Technology) Program
• …an assessment tool to test hypothesized outcomes with
Develop students through one-on-one interviews
• …a pilot study testing the assessment tool across a sample
Execute of IBT and non-IBT students (N>100)
• …preliminary findings from pilot study assessing areas of high,
Review moderate and low impact ; make recommendations for future
impact measurement activity

4. METHODOLOGY

5. The Big Picture
• An example of what impact assessment aims to measures and achieve
Constant Feedback Loop
Institution- System-
Individual
Input Output Wide wide
Impact
Impact Impact
Newly Well- More Improved Increase in the
revamped trained , engaged scores on population’s
teacher energized students, standardized job-readiness
training and resulting in tests as a result of a
program motivated higher higher
teachers attendance proportion of
high school
Impact assessment is a method of gaining constant feedback graduates
continuing on
on the design of your organization’s programs with the aim of
to tertiary
refining them in real-time to achieve the results you want education

6. Defining Outcomes and Metrics
 The following expected student outcomes (near- and long-
term) as a result of participation in the IBT program were
identified during the initial phase of the study (slide 1 of 2):
Educational Achievement: Skills Attainment:
• Improved school attendance • Writing skills
• Improved performance in • Organizational skills
regular curriculum • Project management skills
• Improved performance in 10th • Problem-solving skills
standard examination • Basic understanding of
• Heightened interest in STEM- economics (e.g. costs, demand)
related learning (science, • Vocational skills (e.g.
technology, engineering, math) engineering, electric wiring)
• Heightened interest in pursuing
further education

7. Defining Outcomes and Metrics
Behavioral Changes: Employability / Value to Society:
• Confidence in self • Career aspirations
• Increase in self-worth / feeling of self- • Ability to apply skills outside of school
reliance • Aspirations to give back to one’s
• Sense of accomplishment community
• Pride in work • Employment status after 2 /4 / 6 years
• Reduced stress related to school / • Increase in family involvement and
examinations commitment to children’s education
• Increased comfort in asking questions • Increase in community involvement and
• Increased comfort in working with commitment to local education
groups (including mixed gender groups) institutions
• Increase in creativity • Improved local economy with better job
• Increase in entrepreneurial spirit opportunities and a well-educated
community
The pilot study has illuminated preliminary findings to support several expected
outcomes across the four defined categories. The analysis, however, will focus on
behavioral outcomes – the least studied outcomes of the IBT program

8. The Assessment
 The assessment begins with each student answering the
following four unaided, open-ended questions in written
form, anonymously.
 The purpose is to achieve: a) an unbiased view of student
preferences of school subjects, specifically in favor of or
against STEM-related courses, and b) students’ unbiased
evaluation of their school and IBT (where applicable)
 Q1: What is your favorite subject in school? Why?
 Q2: What is your least favorite subject in school? Why?
 Q3: What do you like most about your school*? Why?
 Q4: What would you change about your school*? Why?
*IBT students were asked specific about the IBT program in Q3 and Q4 to gain a deeper understanding of student
perceptions of the program and its benefits

9. The Assessment
 One-one-one interviews with each student and ethnographic
observation are the core of the assessment. The interviews combine
qualitative and quantitative measures (complete assessment tool can be
found in the appendix)
 The following is an excerpt from the tool where the students rate the
importance of specific skills learned at school and within the IBT program.
The purpose is to measure where IBT and non-IBT students differ in their
perception of skills attainment, and which specific skillsets emerge as the
most significant for students in the IBT program
Q1: Please read the following 14 skill areas. Which FIVE do you feel are
the most important skills you are learning at school?
Study skills Creativity Reading/writing Math
Science Communication skills Computer skills Career planning skills
Time mgmt. skills Business skills Organizational skills Problem solving skills
Job-specific skills Teamwork skills

10. The Assessment
 The tool also includes an 11-question self-esteem battery, adapted from the
widely accepted self-esteem scale developed by Dr. Morris Rosenberg*.
Students rate their level of agreement with each statement (below) on a 4-
point scale from strongly agree to strongly disagree.
 The questions aim to gain insight into key behavioral changes among IBT
students.
I am proud of my projects and homework I enjoy working in groups
I am proud of my marks at school I enjoy working in mixed groups (boys and girls,
combined)
I feel positive that I will pass the 10th standard I like to ask questions to my teachers
examination
I am excited about my future I feel that I make my parents / family proud
I feel that I will have great opportunities after I If there is a problem, I believe I can fix it
finish school
I feel that I am gaining many good skills at school
*Source: http://www.bsos.umd.edu/socy/research/rosenberg.htm

11. Sampling and Administration
 During the pilot study, data was collected from 109 students across
four secondary schools in rural Maharashtra. Survey counts by
gender and category are below:
Survey Sample: Impact Assessment Pilot Study
Total sample: 109 students
IBT: 60 students (55%)
Non-IBT (control group): 49 students (45%)
9th Standard: 57 students (52%)
10th Standard: 52 students (48%)
Male: 61 students (56%)
Female: 48 students (44%)
 The surveys were administered in Marathi, the students’ local
language with a duration of 5 -10 minutes.

12. PRELIMINARY
FINDINGS

13. Preliminary Findings
 Most important skills learned through school / IBT
Top 5 (in order of frequency) Most Selected Skills
IBT Students Non-IBT Students
Study skills Study skills
Teamwork skills Reading/writing
Problem-solving skills Problem-solving skills
Career-planning skills Science
Business skills Computer skills
IBT students are more likely to prioritize outcomes-focused, non-
academic skills such as business skills, teamwork, and career-
planning skills – all core foci of the IBT curriculum

14. Preliminary Findings
 Effect of IBT Program on Importance Placed on Skill Attainment
(slide 1 of 2)
Skill Control % IBT % % Effect
Selected Selected
Career planning skills 20% 43% 112%
Time management skills 27% 18% (-31%)
Business skills 33% 42% 28%
Organizational skills 24% 13% (-46%)
Problem solving skills 42% 43% 1%
Job-specific skills 30% 33% 9%
Teamwork skills 41% 45% 10%
Highlighted skills are those that IBT students chose more often than non-IBT
students as being a Top 5 important skill learned at school.

15. Preliminary Findings
 Effect of IBT Program on Importance Placed on Skill Attainment
(slide 2 of 2)
Skill Control % IBT % % Effect
Selected Selected
Study skills 71% 70% (-2%)
Creativity 14% 22% 52%
Reading/Writing 53% 37% (-31%)
Math 41% 38% (-6%)
Science 43% 40% (-7%)
Communication skills 18% 15% (-18%)
Computer skills 43% 38% (-11%)
Negative effect does not indicate reduced impact on specific skill areas; instead, it
is a reflection on the prioritization given by students on other skills

16. Preliminary Findings
 By overlaying the self-esteem question set with the proposed
outcomes of the IBT program, we identified 6 key behavioral
success factors to assess:
I like to I am
ask excited
I am questions about my
proud of
future If there is
my I feel a problem,
projects positive
I am I can fix it
that I
Do you proud of
will
want to my grades I feel like I
pass I like working
start your I have have many
great in groups (and
own learned
opps. mixed-gender
business? good skills groups)
Measures and indicators to assess factors

17. Preliminary Findings
Measured Outcome Control IBT %
on a 100 pt
scale Score Score Difference
Entrepreneurial
Spirit 65.3% 71.7% 9.7%
Factors Most Affected Among
Self-Pride /
3.61 3.67 1.6%
Select Survey Questions
Self-Worth
Self-Esteem Battery +
IBT Students
Self-Confidence 3.58 3.66 2.1%
Personal
Aspirations 3.51 3.40 -3.1%
Self-Reliance 3.51 3.58 2.1%
Interpersonal
Skills 3.41 3.58 4.9%

18. Preliminary Findings
Learnings / Recommendations
• Pride in one’s work • Group work and practicals, “learning by
Areas of • Recognition of one’s doing,” has had the biggest impact on
High achievements students’ confidence and interpersonal skills
• Identify how these areas can be leveraged to
Impact • Entrepreneurial spirit
positively influence students’ long-term view
• Gender equality and development
• Confidence in test • Self-reliance (the ability to solve problems
Areas of taking and think independently) is a core expected
Moderate • Confidence in asking outcome of IBT and should be emphasized
Impact questions more during the program; same for asking
questions, where many students are still
• Self-reliance hesitant
• Family pride in child’s • Connect more with parents and families
Areas of accomplishments about what IBT is offering their children.
Low/No This will not only positively affect the
• Perception of a bright
Impact future
families and the students’ self-esteems, but
will help in getting families to contribute fees

19. Preliminary Findings
 Student Aspirations for Continuing Education
Q: After 10th standard, what are you interesting in doing?
Control IBT
Go to ITI or another technical 18% 22%
school to earn my diploma
Attend junior college (11th and 12th) 78% 74%
Go work with my family 0% 2%
Go work somewhere else 0% 0%
I don’t know 4% 2%
While IBT students are slightly more interested than their peers in pursuing technical
education after 10th standard, the majority are still interested in junior college.
Recommendation:
LAHI should evaluate its position as a feeder into diploma programs and
how this incentive is being communicated to IBT students and families

20. WRAP-UP

21. Key Findings From Pilot Study
LAHI has created positive impact for IBT students across a number of
measures. In this study, the most notable outcomes are:
Transmitting the
Inspiring Future Facilitating Strong
Importance of Life
Entrepreneurs Interpersonal Skills
Skills Learning
• IBT students are twice • Specifically, an IBT • IBT students are more
as likely as their peers student is 10% more likely than their peers
to view career likely to be interested to feel comfortable
planning as a in starting his or her working in groups,
fundamental skill own shop or business particular mixed
learned during their • IBT students’ ideas range gender groups (7%
secondary education from tailoring and welding effect)
• Business, problem-solving, shops to computer • This is a significant
and job-specific skills are institutes and medical achievement in schools
also perceived to be of clinics where rote learning and
higher importance among individual achievement are
IBT students the norms

22. Recommendations For Lend-A-Hand India
In areas of moderate and
In areas of high impact…
low impact…
Identify drivers (e.g. teaching Pinpoint gaps between expected
styles, level of practicals) of and actual outcomes (e.g. students’
student achievement that can comfort in asking questions) and
be leveraged towards improving determine programmatic changes
on other measures that can be tested to minimize
achievement gaps
Questions to consider:
 What outcomes exceeded expectations?
 Which measures were less impacted by the program than what was
expected?
 How can LAHI leverage areas of high impact (e.g. entrepreneurial
drive) to positively affect areas of moderate (e.g. creativity) and low
impact (e.g. family pride)

23. Recommendations For Further Impact Work
Impact measurement is not a static, annual process to appease funders; it is a
dynamic, continuous effort aimed at understanding, refining and improving
upon an organization’s programs and outcomes
 Be aware of measurement pitfalls*, such as:
 Trying to assess a whole program instead of specific components
 Aspiring for scientific-level design that is appropriate for large scale
populations when striving for generalizable trends but not for
context-specific learning where a finer, more customized and
qualitative approach often yields richer results
 Focusing on uncontrollable , long-term community impact metrics as
a measure of a program’s success over tangible, measureable, near-
term outcomes
*R&D: A New Form of Evaluation, TCC Group

24. Recommendations For Further Impact Work
 Expanding on the pilot study, future steps should
include:
 Communicating expected outcomes and measurement goals to
IBT schools to onboard key stakeholders (e.g. headmaster,
instructors) to a more robust assessment system
 Tracking outcomes by standard to understand when and how
outcomes evolve over program duration
 Scaling assessment across all IBT programs with the aim of
identifying and replicating star school strategies for
maximizing outcomes
 Expanding the next phase of the study to include more control
variables such as family involvement, family income level,
academic achievement, etc.

25. APPENDIX

26. Student Verbatims – Positive Reflections on IBT
 “I will always have the opportunity to do something because of IBT”
 “Because of IBT, we can get jobs anywhere from nursing to engineering”
 “I like IBT because we get knowledge of different things”
 “I chose IBT because if I don’t get a proper job, I can start my own business”
 “We get to learn many different and new things; IBT taught me how to make use
of waste”
 “I get to know the benefits of IBT because in our future, we can do wiring, prepare
chikki, and first aid solutions on our own. IBT is going to help us a lot more in our
future.”
 “Even if I cannot take further education, I can do something or the other and
make a living because of what I’ve learned through IBT”
 With the help of IBT, you learn how to solve problems and start a small-scale
business in the future. “
 “We get to learn many things that can help our family.”
 “It’s very beneficial and it’s the only place where girls and boys can work
together”

27. Student Verbatims – Suggestions for IBT
 “We should be taught about computers as well”
 “Internet should be added as the 5th section of IBT”
 “Each student should be required to do each practicals because
some don’t do it”
 “Practicals should be increased”
 “IBT should double its syllabus and newer things should be
taught”
 “Sometimes materials are in shortage and when tools get broken,
it takes long to get them fixed”
 “An electronic switchboard should be provided in all four
sections”
 “There should be more developed machineries for the practicals
so we can learn more things”

28. IBT Impact Assessment Tool (slide 1 of 4)
Pre-screener questions:
 School
 Standard
 Gender
 IBT Status (IBT or Non-IBT)
Self-reported questionnaire:
 Q1: What is your favorite subject in school? Why?
 Q2: What is your least favorite subject in school?
Why?
 Q3: What do you like most about your school*? Why?
 Q4: What would you change about your school*? Why?
*IBT students were asked specific about the IBT program in Q3 and Q4 to gain a deeper understanding of student
perceptions of the program and its benefits

29. IBT Impact Assessment Tool (slide 2 of 4)
Interview questions:
 Q5: Please read the following 14 skill areas. Which FIVE do
you feel are the most important skills you are learning at
school?
A) Study skills B) Creativity C) Reading/Writing D) Math E)
Science F) Communication skills G) Computer skills H) Career
planning skills I) Time management skills J) Business skills K)
Organizational skills L) Problem solving skills M) Job-specific skills N)
Teamwork skills
 Q6: After 10th standard, what do you want to do? A) Go to ITI
or other school to get a diploma B) Go to Junior College C) Go work
with my family D) Go work somewhere else E) I don’t know

30. IBT Impact Assessment Tool (slide 3 of 4)
 Q7: What is your dream job?__________________
 Q8: Why?_______________________________
 Q9: Do you ever want to start your own shop or
business? A) Yes B) No
 Q10: What kind of shop or
business?________________________________
 Q11: Please reach each statement carefully. For each
statement, tell me if you: A) Strongly agree B) Agree
C) Disagree D) Strongly Disagree
 I am proud of my projects and homework
 I am proud of my marks at school
 I feel positive that I will pass the 10th standard examination

31. IBT Impact Assessment Tool (slide 4 of 4)
 Q11 (continued): Please reach each statement carefully.
For each statement, tell me if you: A) Strongly agree
B) Agree C) Disagree D) Strongly Disagree
 I am excited about my future
 I feel that I can have great opportunities after I finish school
 I feel that I have gained many good skills at school
 I enjoy working in groups
 I enjoy working with mixed groups (boys and girls)
 I like to ask questions to my teachers
 I feel that I make my parents / family proud
 If there is a problem, I believe I can fix it