Evaluation Workshop

1. Using evaluation to measure impact
Jean Taylor, Strategy and Development Manager, Lambeth Council

2. Session coverage and learning objective
This session will:
• introduce evaluation and its benefits;
• offer guidance on how to choose the right evaluation approach;
• describe the main stages of evaluation; and,
• key issues you need to consider at each of these.
Learning objective:
• Delegates will gain an understanding of when to use evaluation,
how to choose an appropriate evaluation approach and key issues
they should consider when undertaking an evaluation

3. What is evaluation?
‘Evaluation is an objective process of
understanding how a policy or other
intervention was implemented, what effects
it had, for whom, how and why’.
The Magenta Book: Guidance for Evaluation, The Cabinet
Office (2011)

4. What are the benefits of evaluation?
• Development/improvement of practice
• Support reinvestment in future activity
• Help save money in future (by helping
people focus on what works)
• Provides accountability: how funding has
been spent, what has been achieved,
assessing return on investment

5. Where evaluation occurs in policy cycle
Rationale
Objectives
Appraisal
and planning
Monitoring
Evaluation
Feedback

6. Factors influencing evaluation approach
• The nature of the policy, the scale of its objectives, complexity,
innovation, form of implementation and future direction;
• The objectives of the evaluation and the types of questions it
would ideally answer;
• The timing of key policy decisions and the information on which
they need to be based;
• The types of impact which are expected, on what timescales, and
the availability of information and data on them;
• The time and resources available for the evaluation.
Adapted from the Magenta Book

7. Three main types of evaluation
• Process: whether policy being implemented as
intended, what works or not
• Impact: objective test of the changes that have
occurred and the extent these are attributable to the
policy
• Economic: compare benefits of the policy with the
costs

8. Impact evaluation
• What were the policy outcomes, what change was
observed and how much, how much could be said to
have been caused by the policy as opposed to other
factors?
• Did the policy achieve its stated objectives?
• How did any changes vary across different individuals,
stakeholders, groups etc and how did this compare with
what was anticipated?
• Did any outcomes occur which were not originally
intended and how significant were they?

9. Types of data
• Existing data
– Monitoring data/performance data
– Service level surveys
– Spontaneous feedback (e.g. via social media)
• New data
– Surveys (e.g. to measure scale of change)
– In-depth interviews (e.g. to understand impacts and factors
underpinning impact)
– Focus groups
– Case studies (e.g. to understand impact in context)
– Other modes: visual evidence

10. Main stages/components of evaluation
1. Defining the policy objectives and intended outcomes
2. Defining the audience for evaluation
3. Identifying the evaluation objectives and research questions
4. Selecting the evaluation approach
5. Identifying the data requirements
6. Identifying the necessary resources and governance arrangements
7. Conducting the evaluation
8. Using and disseminating the evaluation findings
Source: The Magenta Book, 2011

11. Main stages/components of evaluation:
NPC’s four pillar approach
1. Map your theory of change
– Provides clarity, reveals causal
links, offers coherent framework
2. Prioritise what you measure
– What is most important (not what is
most convenient)
3. Choose your level of evidence
– Depends on your needs,
capabilities, resources (and those
of your audience)
4. Select your sources and tools
– Existing tools or data sources,
bespoke ones

12. Logic and theory of change models
• Way of thinking through:
– the issue you are trying to address
– the change you want to achieve
– the steps you need to take to get there
• Not a new way of thinking
– It is a way of formalising your thinking
• Distinction between logic model and ToC = latter
surfaces assumptions about interdependencies

13. Key components
• Range of different approaches to ToC but all
include following components:
– Definition of final outcome/goal(s)
– ‘Backwards mapping’ – mapping of intermediate
outcomes that need to be achieved on way to final
outcome
– Definition of activities that will make change happen
• In contrast to logic models, theory of change
surfaces assumptions about what will make
change happen

14. Example Theory of Change:
A Charities Evaluation Service Planning
Triangle© for a supported housing project

15. Example of
Theory of
change: logic
model
Source: NPC

16. PRACTICE SESSION

17. Deciding your approach: example
projects
• Read the brief project description and then as a group develop a
proposed evaluation approach. Please answer these questions:
– What is the evaluation for and who are the audience?
– What are the key outcomes to be measured?
– What should the evaluation focus on (priority measures)?
– What level/quality of evidence is needed?
– What sources of information and data will you draw on and what
tools will you use for data collection?
– When you would undertake the data collection?
– What other information would you ideally have about these
projects?

18. Further resources
The Magenta Book: Guidance on Evaluation, The Cabinet Office (2011)
https://www.gov.uk/government/uploads/system/uploads/attachment_d
ata/file/220542/magenta_book_combined.pdf
Building Your Measurement Framework: NPC’s Four Pillar Approach
(2014)
http://www.thinknpc.org/publications/npcs-four-pillar-approach/
Creating your theory of change: NPC’s practical guide (2014)
http://www.thinknpc.org/publications/creating-your-theory-of-change/

19. Check out the Insight Hub
What we know about
trends in our services
What we know about
who we support
What we know about
our customer service
transactions
What we know about
issues reported to us
How Lambeth
compares
What are the unit
costs & benefits of
services
http://bit.ly/insighthublambeth
http://bit.ly/insighthublambethgov

20. Come to our next sessions
Sign up at http://bit.ly/localinsightprog
Theme Date
Calculating social impact 17 May Community Action
Network
Understanding what matters to local residents 24 May Lambeth
Understanding the strengths & needs of local
communities
1 June Young Foundation
Understanding local demographics 7 June LGA
Using research to inform policy 14
June
Newham Council