Goa Escorts WhatsApp Number South Goa Call Girl … 8588052666…
TTI PEC Nairobi Workshop - Unpacking impact and influence
1. Unpacking Impact and Influence:
Approaches to Monitoring & Evaluating
April 28, 2014
TTI PEC Africa workshop - Nairobi
Courtney Tolmie (Results for Development Institute)
2. A few notes to start
This session focuses on Organization M&E. That said, most of the
lessons can be scaled down to the project level.
You will guide the session – and giving real time feedback:
2 | R4D.org
Way too fast.
Please slow
down.
Just right.
You can
speed up.
3. How do we think about Impact and Influence?
1. What IMPACT does your organization seek to have?
2. What and who do you work to INFLUENCE as an organization?
3. What is the link between IMPACT and INFLUENCE in the work
that you do?
3 | R4D.org
4. Monitoring and Evaluation – the link to Impact and
Influence
In other words, a way of following whether you are doing what you
planned to do and whether you are achieving the IMPACT you wanted
to, including Policy INFLUENCE.
4 | R4D.org
Monitoring: a continuing function
that aims primarily to provide the
management and main
stakeholders of an ongoing
intervention with early
indications of progress, or lack
thereof, in the achievement of
results.
Evaluation: systematic and objective
assessment of an on-going or completed
project, program, or policy, and its
design, implementation and results. The
aim is to determine the relevance and
fulfillment of objectives, development
efficiency, effectiveness, impact, and
sustainability.
Definitions from the World Bank -
http://siteresources.worldbank.org/INTBELARUS/Resources/M&E.pdf
5. What Makes an organization-level M&E System Effective?
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
5 | R4D.org
6. M&E Systems – Defining a Theory of Change
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
6 | R4D.org
Defining a Theory of Change
Clarifying your organization
objectives and desired
impacts
Mapping out what is
required to achieve these
impacts
Linking to the activities and
actions your organization
will take to achieve the
impact
Also highlighting
assumptions (including
context issues)
Similar structures -
logframes
7. M&E Systems – Defining Indicators
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
7 | R4D.org
Defining Indicators
Turning what you hope to
achieve into what you can
actually measure
Making indicators SMART
Value (and tradeoff) in
collecting indicators on
impact and outputs
8. M&E Systems – Designing Methods
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
8 | R4D.org
Defining and designing Methods
Identifying how to actually
collect data on the
indicators that you have
defined
Timing – monitoring,
evaluating or both
Things to consider –
capacity, resources, what
you are trying to achieve
9. M&E Systems – Ensuring Feedback Loops
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
9 | R4D.org
Ensuring Feedback Loops
The most forgotten step of
M&E systems
Results for M&E can help
improve programs and
ultimately impact
Best to have formal
structure to build results
back into future work
Moving from culture of
M&E to MEL
10. Mapping areas for development
Based on your existing M&E system, assess how you are doing in
each element of effective M&E
Scale from “we don’t do this at all” to “we do this very well on all
areas of work”
10 minutes to map your organization – then we will come back to
review
10 | R4D.org
12. Defining Theories of Change
Examples from other organizations
Tips from the group about how to do it well
Ideas for moving forward
12 | R4D.org
13. 13 | R4D.org
Build evidence on
better ways to
budget for higher
education
Share
recommendations
with Minster of
Education and
Finance
New budget passed
with
recommendation
It would be easier if organization TOCs looked like this:
But this is not realistic
WHY NOT?
• TTs are trying to achieve bigger changes than just this
• A single TT is going to likely need to take a lot of different actions to
get to that end impact
• A single TT is not going to individually achieve this impact
16. Note about Theories of Change
A good TOC for influencing policy is incredibly complex.
Your organization may not try to affect all parts of the TOC – and
thus may not choose to do M&E for all factors that go into policy
influence.
But good to be aware.
16
17. Defining Theories of Change
Examples from other organizations
Tips from the group about how to do it well
Ideas for moving forward
17 | R4D.org
18. Defining Theories of Change – TOC Pitfalls to avoid from
Stanford Social Science Review
1. Confusing accountability with hope
2. Creating a mirror instead of a target
3. Failing to take external context into account
4. Not confirming the plausibility of your theory
5. Creating a theory that is not measurable
6. Assuming you have figured it all out
18 | R4D.orghttp://www.ssireview.org/blog/entry/six_theory_of_change_pitfalls_to_avoid
What lessons can you share from your organizations –
things that have worked and what has not worked?
20. Defining Indicators
What types of indicators work well, especially for measuring
policy influence and impact?
Tips from the group about how to do it well
Ideas for moving forward
20 | R4D.org
21. Examples of good indicators – for the full organization
While the focus of this program is on PEC, it is worth noting that
there are multiple components of a think tank’s performance worth
monitoring:
21 | R4D.org
From Raymond Struyk’s forthcoming book – Improving Think Tank Management
Public
Policy
Perspective
Funder
Perspective
Internal
Business
Perspective
Innovation &
Learning
Perspective
Financial
Perspective
22. Examples of good indicators – for the full organization
While the focus of this program is on PEC, it is worth noting that
there are multiple components of a think tank’s performance worth
monitoring:
22 | R4D.org
From Raymond Struyk’s forthcoming book – Improving Think Tank Management
Public
Policy
Perspective
Funder
Perspective
Internal
Business
Perspective
Innovation &
Learning
Perspective
Financial
Perspective
23. Examples of good indicators for PEC impact and influence
Some helpful indicators from the center for International Governance Innovations:
23 | R4D.org
From www.cigionline.org/blogs/tank-treads/communications-and-impact-metrics-think-tanks
Indicators of Exposure:
• Media mentions
• # and type of publications
• Scholarly citations
• Government citations
• Think tank ratings
Indicators of Resources:
• Quality, diversity, stability of funding
• Characteristics of researchers
• Quality, # of networks, partnerships
Indicators of Demand:
• Events
• Digital traffic and engagement
• Access to officials
• Publications sold, downloaded
Indicators of
Policy Impact &
Quality of Work:
• Policy
recommendations
considered or
adopted
• Testimonials
• Quality of think tank
work
24. Other indicators we have seen used
24 | R4D.org
If the objective is …
Specific policy
recommendation
implemented
Change in the debate
… Possible indicators are:
Invitation to share findings with specific audiences
(government, others)
Attendance of decision makers at dissemination
events
(if available) Information on issue being debated in
policy discussions
Request for research by policymakers on the new
topics
Greater debate on topic in media
(if available) Information on issue being debated in
policy discussions
Longer term – access
to policymakers
Individual meetings with policymakers
25. A note about impact indicators …
In general, these are not going to be attributable.
Attribution is difficult
Almost without exception, it takes a village.
This does not mean impact indicators are not helpful – especially if
seeking to improve results.
25 | R4D.org
26. Defining Indicators
What types of indicators work well, especially for measuring policy
influence and impact?
Tips from the group about how to do it well
Ideas for moving forward
26 | R4D.org
27. Defining Indicators
What types of indicators work well, especially for measuring policy
influence and impact?
Tips from the group about how to do it well
Ideas for moving forward
27 | R4D.org
29. Designing methods for M&E
What common options are others using?
Tips from the group about how to do it well
Ideas for moving forward
29 | R4D.org
30. Options for M&E approaches
30 | R4D.orgTable from ODI Background Note – A guide to monitoring and evaluating policy influence (Harry Jones)
Does not include common impact evaluation methods – RCTs,
process tracing, etc.
31. Designing methods for M&E
What common options are others using?
Tips from the group about how to do it well
Ideas for moving forward
31 | R4D.org
32. Designing methods for M&E
What common options are others using?
Tips from the group about how to do it well
Ideas for moving forward
32 | R4D.org
34. Ensuring feedback loops
Do results feed back into the think tank automatically?
Tips from the group about how to do it well
Ideas for moving forward
34 | R4D.org
35. Connecting M&E Results to Think Tank Changes
Two potential reasons to undertake M&E:
To achieve the second, M&E can’t stop at data analysis – it needs to
feed back into the think tank.
35 | R4D.org
ACCOUNTABILITY to
funders and others
BETTER RESULTS for think
tank impact and influence
36. Questions to ask to design a strong Feedback Loop
(Knowledge Management)
36 | R4D.org
The Question
WHO should hear the
M&E results?
WHEN should they
get the information?
The Answer
(1) Who is the best position to exploit the
information – capacity and interest
(2) More generally, anyone who SHOULD know
(1) At a time when they can act – capacity and
opportunity
(2) Generally more than once
WHAT should they
receive?
(1) Format – useful but not necessarily fancy
(2) Ideally comparisons to baseline data
In general, the Feedback Loop is more likely to be effective if designed early –
proactive rather than reactive.
Built from Raymond Struyk’s forthcoming book – Improving Think Tank Management
37. Ensuring feedback loops
Do results feed back into the think tank automatically?
Tips from the group about how to do it well
Ideas for moving forward
37 | R4D.org
38. Ensuring feedback loops
Do results feed back into the think tank automatically?
Tips from the group about how to do it well
Ideas for moving forward
38 | R4D.org
40. Conclusion
M&E is not easy – and requires capacity. This is especially true for
organization-level M&E and for policy influence M&E.
Match M&E plan to capacity
Moving from M&E to MEL (L = “learning”)
Linking the two objectives of M&E – accountability and better results
Some methods and indicators can address both – where
possible, don’t double your work
Example – how can you use the IDRC monitoring indicators for
PEC to help you achieve better results?
40 | R4D.org
41. Other Great Resources (in addition to those referenced on
this powerpoint)
ODI work on RAPID
On Think Tanks page on Monitoring and Evaluating Influence
(http://onthinktanks.org/topic-pages/topic-page-monitoring-and-
evaluating-influence/)
Research 2 Action PEC page
41 | R4D.org