The monitoring, evaluation and learning design of the MasterCard Foundation Scholars Program. Theory of Change, logic model and learning questions. Design challenges and solutions.
Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.
1. Learning Across Borders:
A Monitoring, Evaluation, and Learning
Framework for The MasterCard
Foundation Scholars Program
March 2014
2. Overview of Presentation
The MasterCard Foundation
The Scholars Program
Monitoring, evaluation, and learning (MEL) framework
MEL challenges, approaches, and lessons learned
2
4. The MasterCard Foundation Scholars Program:
Context and Programmatic Need
Developing countries have achieved a remarkable expansion
in educational enrolment, however:
Enrolment in secondary and tertiary education remains low, particularly
among poor and rural populations
There are concerns about access to quality and relevant education,
even as demand for an educated workforce increases
Many skilled graduates migrate to higher-income countries for better
remuneration and working conditions
International scholarship programs too often contribute to brain drain
4
5. The MasterCard Foundation Scholars Program
$500 million, 10-year initiative to educate and support 15,000 young
people – particularly from Africa – to act as “socially transformative
leaders,” driving change and making a positive social impact in their
communities.
11,000 secondary level Scholarships, 6,000 tertiary
Target population: youth who have demonstrated academic talent,
social consciousness and an orientation toward leadership, and who
come from highly disadvantaged socioeconomic backgrounds.
Implemented with and through a global network of 20 NGOs and
secondary and tertiary education institutions, who select and directly
support the Scholars.
5
6. The MasterCard Foundation Scholars Program:
Program Components
6
Comprehensive scholarships
Transition support
21st-century skill building
Give-back component
Alumni network
9. Evaluation and Learning in the Scholars Program
Objectives:
Continuous program quality improvement
Evidence of outcomes, impact and best practices…
For our partners
For the Foundation’s strategic planning
For the global community of practitioners
As the basis of a proactive argument for propagating successful
Program elements and the overall vision of the Program.
9
10. Evaluation and Learning Overview
Program-wide evaluation: primary end-point 2016
Implementation study
Impact study
Institutional change study
Longitudinal studies of outcomes and impact
Outcome study (subset of Scholar cohorts and comparison groups): continued past 5-year point
Outcome tracking (all cohorts)
Anticipated outcomes (Stay in / return to Africa, Give-back, career and leadership outcomes,
citizenship outcomes)
Long-term qualitative work to understand Scholars’ roles in promoting social and economic
development
Partner-initiated Research
Rapid studies on emergent issues, aimed at providing actionable intelligence.
10
13. Approach
and Inputs
Activities Outputs ImpactsOutcomes
Inputs
Partner
expertise
To Serve Scholars
Academic and
other support
Approach
Provide
Scholars with a
wide range of
opportunities
for educational
support and
enrichment,
give-back, and
leadership
development
At Institutions
Revise recruitment
procedures
Scholars using
academic and
other support
services
Partners
disseminate
lessons learned
Increased
academic
performance
Institutionalized
changes foster
best practices
New cohort of
leaders
supports
social
transformation
and economic
growth,
particularly in
Africa
13
The MEL Framework: Logic Model Components
14. The MEL Framework:
Learning Questions, Methods, and Data Sources
Broad Learning Questions Main Design/Methods Data Sources
Implementation
How was the Scholars
Program implemented?
Mixed methods
Qualitative data,
complemented by
quantitative data
Impacts
In particular, what is the
impact of the Program on
educational, employment,
go-back, and give-
back outcomes?
Comparison group design
• Random assignment
• Regression discontinuity
• Propensity score matching
Survey data, complemented
by qualitative data
Scale-up
Did the Scholars Program
become a catalyst for
broader change within
and beyond partner
institutions?
Qualitative analysis
Qualitative data,
complemented by
quantitative data
14
16. Challenges, Solutions, and Lessons Learned:
Designing a Unified MEL Framework
Challenge
Develop a single unified MEL framework to evaluate The MasterCard
Foundation Scholars Program
Solutions
Develop the MEL framework from the ground up through a thorough
understanding of partner projects and what they have in common
Use a flexible approach that allows for different methods
for each partner: Partner-specific comparison group designs allow for a
rigorous evaluation while using different methods to draw the
comparison group
Random assignment
Regression discontinuity
Matched comparison group (via propensity score methods)
16
17. Challenges, Solutions, and Lessons Learned:
Measurement
Challenge
Locating applicants based on application data may be challenging
Several key outcomes are difficult to measure
Key outcomes, including educational attainment, career and service
outcomes, giving back and going back, occur several years in the future
Solutions
To locate hard-to-find applicants, use their own and their schools’
contact information
Draw survey items from existing instruments and pilot our
own instruments
Establish a plan for tracking The MasterCard Foundation Scholars over
time that is appropriate to the context
17
18. Challenges, Solutions, and Lessons Learned:
Measurement—Baseline Data Collection in Uganda
Sample: 823 applicants spread
around Uganda
Applicants were located by calling
the phone number provided
in their application
This is the first of three rounds of data
collection planned
18
92%
8%
Locating applicants
Located
Not located
82%
18%
“Give-back” attitudes:
Helping people in need is
something one should do…
for anyone
for friends and
family
2% 15%
83%
Contact information
No phone
number
1 phone
number
2 phone
numbers
19. Challenge
Incorporate formative evaluation components that will generate
relevant feedback for partners throughout the evaluation
Early feedback can then inform mid-course corrections as needed
Solutions
Engage with partners while developing MEL activities
Collect quantitative and qualitative data at regular intervals
Provide partners with feedback from all data collection efforts
19
Challenges, Solutions, and Lessons Learned:
Blending Formative, Summative Evaluation Approaches
20. Phone interviews with 18 graduating tertiary Scholars
Opportunity to gain early information on Scholar experiences
This early outcome information will support continuous quality
improvement
Scholars offer suggestions for Program improvement
Increase flexibility in resource allocation—for example, could leftover
resources for textbooks be used for transportation?
Coordinate Scholar attendance at academic conferences
to facilitate networking
Provide orientation to the Program in one intensive session early on,
rather than spread over several sessions
20
Challenges, Solutions, and Lessons Learned:
Regular, Relevant Feedback to Partners
21. Challenges, Solutions, and Lessons Learned:
Regular, Relevant Feedback to Partners
(continued)
Scholars also say what is going well
The Program gives Scholars “the flexibility, time
and resources to actually do the work you set out
to do”
“[I’m] trying my best to really use all the
knowledge they’ve given to me and trying to
implement it in myself first and then try to
introduce it to society”
The Masters in Development program is a “dream
program” that fits well with career goal
of reducing poverty in her home country
21
22. Conclusions
We have designed MEL activities
that respond to the complex nature
of The MasterCard Foundation Scholars
Program with an approach that is
• Flexible, yet rigorous
• Capable of measuring key outcomes now
and in the future
• Formative and summative in nature
• Collaborative and inclusive, responding
to needs across the partnership
22