AEA Conference Presentation, October 2012 Originally developed by: Dr. Helzi Noponen Presented by: Anna Kiel Martin Katrina Mitchell

1. Self Help Group Promotion, PRADAN. Photo source: Helzi Noponen
THE INTERNAL
LEARNING SYSTEM
ASSESSING IMPACT WHILE EMPOWERING
INTERNAL LEARNING
Anna Kiel Martin American Evaluators Association National Conference, Minneapolis
Katrina Mitchell October 26, 2012

2. WHAT IS ILS?
• SIMPLE
• Quantitative
• Evaluation +
• Non-extractive
• Highly participatory
• Pictorial/accessible
• Customized
• Embedded in the program
• Multi-year

3. HOW DID ILS DEVELOP?
Helzi Noponen in Bihar, 2006. Photo credit: Valerie Arendt

4. WHY USE ILS?

5. EXERCISE:
HOW DOES THIS WORK?
You have been given a sample page.
Identify what the purpose of the page is and how you would
interact with it as a participant.
How would you use it to assess impact or empower learning?

6. EVALUATION
Sanitation Social Problems

7. LEARNING
“Suicide is Not a Solution”
?

8. PLANNING
Livelihood Production Plan Problem Sorting
?
HPWDS

9. PLANNING
Strategic Interests Plan
?

10. WHAT’S IN THE ILS TOOLBOX?
“Women own the ILS book. It is their record.
Though illiterate, they can “read” it. No one
has ever before suggested they can do such
work. They take the responsibility seriously.
It is like a green light for them to dare to think
about achieving a better life.
Each indicator picture has an implied
program value attached to it.
Women absorb this.
It gives permission or space to discuss
sensitive empowerment and wider social
issues and the courage to attempt change”
Helzi Noponen

11. HOW DO YOU KNOW YOUR
CURRENT SITUATION?
General Health Care Practices
Evaluation formats:
• Yes/no questions
• Quantity questions
• Multiple choice questions
Woman goes to doctor when needed. Safe water practices
• Ranking and/or scaling questions
Knows oral re-hydration therapy. Washes hands after latrine and before
food handling.
Woman herself uses mosquito net.
Hygiene around menses

12. HOW DO YOU IMAGINE
A BETTER FUTURE?
Learning formats: “Suicide is Not a Solution”
• Bad Scene / Good Scenes
• Cautionary Cartoons / Illustrated
Folk Tales
• Dream Scenes
• Panorama Scenes
• 6 panel Stories (photo novella,
picture parade, comic strip)
?

13. HOW WILL YOU CREATE
A BETTER FUTURE?
Plan for Meeting Strategic Interests
Planning formats:
• Problem Sorting Exercises think discuss husband family help friend help get comfort
• Goal Setting Exercises
• Priority Choosing Exercises
other action get service
• Planning Formats – budgeting meet officials work with men
trees, loan repayment plans
write officials research
elite help campaign
0 1 2 3
cluster help SHG help
other SHG help protest action exposure visit speak out training

14. HOW DO YOU WEAVE IT
TOGETHER?
Planning
Learning
Evaluation

15. EXERCISE:
HOW DO YOU…
You are going to a pot-luck and have to bring a dish.
What mental steps do you go through from the invitation to
arriving at the right place, at the right time, with the right food to
share?

16. PARTICIPANT:
HOW DOES ILS CHANGE BEHAVIOR?
1. Collect
2. Assess
3. Analyze
5 Tasks
4. Plan or alter
5. Document, share, reinforce

17. PARTICIPANT: HOW DO I USE THIS
TO IMPROVE MY SITUATION?
Dalit women in Varanasi displaying their job cards Photo source: Leena Patel/UN Women (seeking permission)

18. HOW DOES A PARTICIPANT
INTERACT WITH ILS?
• 5 tasks of ILS
• Ownership of diary/
data – participatory
• Census – they all
have one
• Community-building,
decreases isolation,
empowers (story
telling)

19. WHO INTERACTS WITH ILS?
PARTICIPANT PROGRAM STAFF
EVALUATOR AGENCY/ORGANIZATION
Program participants with diaries/workbooks PRADAN, Photo source: Helzi Noponen Field promoter with PRADAN, Photo source: PRADAN
Helzi Noponen in Bihar, 2006. Photo credit: Valerie Arendt Image source: PRADAN website

20. HOW DOES A PROGRAM MANAGER
INTERACT WITH ILS?
• Facilitation in a
group setting
• Reveals conditions/
behaviors which
aren’t readily visible
• Highlights patterns
• Demands radical
accountability
• Encourages
reflective practice
and innovation

21. HOW DOES AN EVALUATOR
INTERACT WITH ILS?
• How to (steps to ILS):
• Design
• Implementation
• Measurement
• Commitment to participatory
techniques
• Effective creation and
implementation mirrors the
tool; lots of learning and
adapting

22. HOW DOES THE AGENCY INTERACT
WITH ILS?
• Impact assessment
• Can quantify hard-to-measure social change outcomes
• Use with outside stakeholders for funding and advocacy
• Commitment to participatory techniques, accountability
and innovation/learning

23. HOW WOULD YOU USE IT?
EARLY CHILDHOOD HEALTH CARE
BUSINESS DEVELOPMENT HOMELESS / YOUTH
Stock photo Maternal / neo natal health care. Photo courtesy of Neighborhood House
Recent immigrants. Photo courtesy of Neighborhood House Stock photo

24. IS ILS RIGHT FOR MY
PROJECT / ORGANIZATION?
YES NO
I want QUANTITATIVE data I need QUALITATIVE data
Learning is important I just need to assess
Participant input is important Summative only
Formative and summative Outcome only
Process and outcome data Short term or one-off evaluation
Longitudinal, long-term
Highly customized
USE “TRADITIONAL” M&E or
EXTRACTIVE PIA

25. HOW WOULD YOU CONSTRUCT
SOMETHING LIKE THIS…
….as it was originally meant to be used?

26. WHAT SHOULD I CONSIDER WHEN
DESIGNING AN ILS DIARY?
Research design
• Identify your needs and analysis plan before you start
Survey design
• Both formative and summative
• Types of questions
• Level of detail
• Relevant to all participants
Diary design
• Pictorialize – lots of considerations!
• Create modules
• Start simple. Build up to complexity and risk.

27. HOW MUCH DETAIL?
DEPENDS…

28. HOW DO YOU TURN PICTURES AND
PENCIL LINES INTO DATA?
PARTICIPANT DIARY INTERVIEWER DIARY
The semi-extractive Land Utilization
Land Utilization
99=not farmer (skip page)
interview event: Grows crop?
0=No; 1=Yes
number of yearly
harvests?
average yield?
base mid point end point
• Side-by-side
number of
harvests
# grain harvests
optimum yield base mid point end point
• Sharing
base mid end
grain yield 5
optimum yield
base mid point end point
• Transparency base mid end
# pulses harvests
base mid point end point
above 75% yield
pulses yield 4
• Verification base mid point end point above 75% yield
# oilseeds harvest
• Human Subjects 50% - 75% yield
base mid end base mid point end point
Protocol
oilseeds yield 3
base mid point end point 50% - 75% yield
# vegetables harvest
base mid end base mid point end point
2
25% - 50% yield vegetables yield
25% - 50% yield
base mid point end point
# fruits harvest
Include sugarcane
base mid point end point 1
base mid end
Less than 25% yield Less than 25% yield
fruits yield
20 20

29. WHAT ARE ITS STRENGTHS?
Traditional M&E Participatory
Impact Assessment
Extractive METHOD Empowering
Donors/policy AUDIENCE Clients
Judging PURPOSE Learning
External ORIENTATION Internal
From Simonowitz, 2001

30. COPYRIGHT, ATTRIBUTION
AND USAGE
A note on copyright, attribution, and appropriate usage.
The Internal Learning System was developed by Dr. Helzi Noponen in partnership with Ford
Foundation India and Imp-Act, and implemented with a number of organizations including
PRADAN, the Handloom Weavers Development Society (HLWDS), ASA, the High-range
Plantation Workers Development Society (HPWDS), PLAN Honduras, and others.
Many of the illustrations contained in this presentation come from diaries/workbooks created
by Dr. Noponen for these organizations.
Parts of this presentation were adapted from presentations and materials
created by Dr. Noponen.
Anna and Katrina are currently working with Dr. Noponen’s family and colleagues to
preserve and extend the legacy of ILS. Please consult with us for attribution and usage
information
prior to distributing or re-using these materials.

31. Muhammad Yunus, Nobel Prize Winner, with Helzi Noponen, 2008. Photo source: Gary Smaby
HELZI NOPONEN, 1953-2012
Innovative mind, open heart.
helzinoponen.wordpress.com

32. THANK YOU!
KATRINA MITCHELL
Research and Strategy for Good
genius@katrinamitchell.com
612-269-2891
ANNA MARTIN
Thinking Evaluation
anna@thinkeval.com
612-805-6460
internallearningsystem.com

33. Building a Diary | Components
The diaries are composed of a series of themed modules. The modules together form a development curriculum that is a logical whole. Each module utilizes a variety
of different approaches, or components, for meeting program goals and evaluation. The three types of components are for learning, planning and evaluation. Every
module does not have all three, but most include at least some evaluation exercises. Usually a learning exercise introduces the module.
Learning Planning Evaluation
Heavily facilitated, typically in a group setting, by Planning tools allow for us to take on greater Some of the evaluation is for impact assessment
the program officer or field agent. These exercises complexity in our lives than if we try to keep it all in (proving) perspective, many of them help the
make up the “curriculum” of the program; they are our heads. The planning exercises in the diary participant to assess their own life and circum-
messages from the program/agency to its constitu- open up a whole new world for participants. For stances in a meaningful way. Many have never
ents about the intervention. They tend to be many, it is the first time they have purposefully thought in these terms before. Simply writing
problem-solving or strengths-based. The purpose created and acted on a plan. These plans help down and quantifying thier life and experience
is to inspire participants on how life could be, or participants move from reactive/crisis orientation to can provoke new ways of doing things, new ways
how a problem could turn out well. They are not a proactive/future orientation. of thinking.
prescriptive but are conversation starters.
It is important that the participant takes an active • Yes/no questions
• Bad Scene / Good Scenes role in their own planning. The diary is a tool to • Quantity questions
• Cautionary Cartoons / Illustrated Folk Tales identify, prioritize and then strategize about the • Multiple choice questions
• Dream Scene problems and possibilities in their life before taking • Ranking and/or scaling questions
• Panorama Scenes (e.g. Gender Benders) action.
• 6 panel Stories (photo novella, picture parade,
comic strip) Planning components also allow for accountability
-- both the participant to themself and the partici-
pant to the group, vice versa.
• Problem Sorting Exercises
• Goal Setting Exercises
• Priority Choosing Exercises
• Planning Formats
Image source: Helzi Noponen
Image source: Helzi Noponen
ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell.
ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com

34. Building a Diary | 5 Tasks of ILS
The core of the ILS strategy is to engage the participants in a five-step process that carries over and informs how they will think about their life. We all do these things,
but the diary makes this process explicit, in order to empower the participants with choice.
1. Collecting Data
Participants own the ILS diary. It is their record. The diary promotes reflection on their current situation, in a struc-
tured and relevant format. Each indicator picture has an implied program value attached to it. The process of
carefully marking and keeping the diary acts as a green light for them to dare to think about achieving a better life.
2. Assessing Data / Change
Participants, though illiterate, can “read” their own diary and that of others. Individually, and in the context of a group, they
gain an understanding of their changing life and livelihood situation. Patterns emerge, the ability to set goals, prioritize
problems and solutions and plan for the future are all facilitated through the use of the diary.
3. Analysing Causes of Change / Troubleshoot
Once a goal is set, or a pattern emerges, participants use their data and assessment of the situation to
identify potential causes and reasons. Asking “why?” and pursuing an answer calls forth the wisdom and
skills of each participant, empowering them to step into a position of primary actor in their lives.
4. Plan or Alter Strategies, Training
With an understanding of the forces at play, participants make intentional changes to their
strategies and behavior for achieving their goals. They begin to track and assess data again,
to understand the impact of this new plan of action—gaining greater self-awareness, confi-
dence and better results in the process.
5. Documenting, sharing and reinforcing values
Sharing between participants can be a wonderful source of inspiration, support and power; discus-
sion is an oft over-looked learning tool.
The diary acts as a testimony to individual experiences, which are often duplicated between group
members. These recorded similarities give permission to discuss sensitive wider social issues and
inequitable structures and the courage to attempt change. Collective action to summon needed
resources and services is bolstered by the diaries’ written record.
The source for all images on this page: Helzi Noponen
ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell.
ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com

35. Why ILS? | Going beyond proving
While ILS can help “prove” program effectiveness, and can be used for internal accountability, its real strength lies in its ability to support improving. It is the learning and
planning components of ILS coupled together with evaluation that make it a powerful tool which is integral to the program design.
Program Impact Assessment Objectives
PROVING IMPROVING / LEARNING
Strengthen Self
Proving to Stakeholder advocacy
program Relationships
participants
and staff
EMPOWERMENT
Planning
Proving to
funders +
organizational Staff Participant
leadership learning learning Reflection +
Assessment
“Traditional” Impact Assessment Participatory Impact Assessment (ILS, and others)
Traditional impact assessment has an external focus (to the donors and Participatory Impact Assessment (PIA) is internally focused, aimed at improving
policy makers) and a more quantitative and “objective” approach. It tends to outcomes through the empowerment of the program staff and participants. PIA falls
be extractive, the information gathered is primarily used outside of the along a spectrum of participatory methods used to gather data from extractive formats
community and little, if any, feedback is provided to the community in terms to fully engaging participants in evaluating their own progress and the program.2 While
they can understand. The purpose of the assessment is to prove effective- some focus on the pictorial aspect of ILS as a way to gather information from a
ness and efficiency rather than to empower the community, participants or low-literacy population, the true strength of ILS is in its abiltity to empower learning and
program staff to learn and improve.1 inspire action.3
1. Marisol Estrella and John Gaventa, “Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review”,as summarized in Anton Simonawitz, “Making impact assessment more participatory” June 2000
2. Anton Simonawitz, “Making impact assessment more participatory” June 2000
3. For a discussion of how ILS differs from PRA techniques using pictures see Noponen, H. (2002) The Internal Learning System—a tool for micro-finance and livelihoods interventions. Development Bulletin 57: 106–110.
ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell.
ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com

36. Building a diary | How to ILS (the basics)
ILS is not just about creating a pictorial diary. It is based on well-grounded research design and starts with the creation of a simple survey instrument using indicators
derived from thorough background research on the community and the program. Initial research is often done with a participatory approach, though a survey could be
designed in collaboration with program staff who have sufficient field expertise. In both cases it is critical to do at least one field test before implementation.
In order to make a diary that is interesting and relevant to all participants, it should contain very few skip patterns. The survey should be designed with questions that
can be answered pictorially such as status or yes/no, simple quantities, multiple choice and scale ratings. Each indicator should relate to a learning component. Do not
include survey questions that are only of interest to program staff or for external proving.
CREATING A DIARY PUTTING IT TO USE
Train program staff
Conduct background research to
develop appropriate indicators and
determine barriers to success Print and mark diaries with ID and date
Distribute census wide
Draft a simple survey instrument From identified barriers and program
including demographics and key goals conceptualize learning and
indicators of program success planning components For baseline make first marks in red
Pictorialize the survey
(Hire an artist to create drawings) ANALYSIS & REPORTING
Likely, issues arise with: Develop research design
Draft layout and assemble learning,
planning and evaluation components a. Layout of a question on a page causes
into modules that form a logical whole. confusion Train interviewers!
b. Individual pictures are not understood at all,
Train 1-2 staff in facilitation or interpreted differently across the group Select sample based on design
c. A particular question or learning component (panel or cross-sectional)
causes discomfort, stone-walling or conflict
FIELD TEST!! from the group Semi-extractive side-by-side
d. Particular subject matter or issues were interview event. Participatory!
repeat Debrief / sharing with overlooked
as necessary participants and staff e. One of the choices of a multiple choice Data coding and entry
question is incorrect or missing
Revise* f. Facilitation of a particular learning component Analysis
should occur differently or at a different time.
Create facilitation guide Report to all stakeholders
ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell.
ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com