National Volunteering Forum - 4 May 2016

1. NATIONAL VOLUNTEERING
FORUM
BIGGER IMPACT AND BETTER
VALUE: DEMONSTRATING THE
IMPACT OF VOLUNTEERING
4 MAY 2016

2. NICK OCKENDEN
HEAD OF RESEARCH
NCVO

3. AGENDA
MORNING SESSION
Time Agenda Item Speaker
10.00am Tea and coffee available
10.30am Welcome and introduction Nick Ockenden
Head of Research, NCVO
10:35am ‘Why is measuring impact important
now?’
Karl Wilding,
Director of Public Policy and Volunteering,
NCVO
11:05am ‘Planning your impact’ Georgina Anstey,
Consultant, NCVO
11.35am Roundtable discussions and
feedback
Chair
12.15pm Lunch

4. AGENDA
AFTERNOON SESSION
Time Agenda item Speaker
1.15pm ‘What difference do your volunteers
make? Top tips on how to undertake a
volunteering impact assessment’
Joanna Stuart,
Senior Research Officer, NCVO
1.45pm ‘Challenges & tools’ Claire Knight, Head of Volunteering,
Breast Cancer Care
Rebecca Stewart, Head of Volunteering,
Age UK
Representing the National Network of
Volunteer Involving Agencies (NNVIA)
2.05pm Q & A Chair
2.20pm ‘Volunteering in care homes’ Matt Hill,
Senior Research Officer, NCVO
2.40pm ‘Research into impact on
health & wellbeing’
Daiga Kamerade,
Work and employment researcher,
University of Birmingham
3.00pm Q & A Chair
3.15pm Closing remarks Chair

5. JOIN THE DISCUSSION
#volforum
@NCVOvolunteers

6. VOLUNTEERING:
WHY IS MEASURING IMPACT
IMPORTANT NOW?
@KARLWILDING
NCVO PUBLIC POLICY & VOLUNTEERING
MAY 2016

7. 1. WHERE ARE WE NOW?
2. TERMS OF TRADE
3. DRIVERS FOR CHANGE
4. CHALLENGES
5. WHERE NEXT?
7

8. 1. WHERE ARE WE NOW?
8

9. VOLUNTEERING

11. 2. TERMS OF TRADE
11

12. WHAT IS IMPACT?
• Impact is overall change we seek to
achieve.
• What we do – our outputs – designed
to deliver impact.
• Impact significantly bigger and more
important than evaluating, or
monitoring, or measuring.

13. 3. DRIVERS FOR CHANGE
13

14. THE BRAVE NEW WORLD OF DOING GOOD

16. 16

18. 18

19. 4. CHALLENGES
19

22. 22

24. WHAT WORKS, EVIDENCE OF HOW
CHANGE HAPPENS

25. 25

26. 26

27. 5. WHERE NEXT?
27

30. PLANNING YOUR IMPACT
GEORGINA ANSTEY
CHARITIES EVALUATION
SERVICE
NCVO

31. PLANNING YOUR IMPACT

32. PLAN
DOASSESS
REVIEW
IMPACT PRACTICE CYCLE

33. WHY PLAN YOUR IMPACT?
 Strategy
 Communication
 Fundraising
 Measurement

34. LANGUAGE AND LOGIC
Resources
used to
create the
services
offered
What is done
with the inputs,
to enable it to
develop and
deliver its
outputs
Products,
services or
facilities that
result from
activities
The changes,
benefits,
learning or
other effects
that result from
what the
project or
organisation
makes, offers or
provides
Broader or
longer-term
effects of a
project’s or
organisation’s
activities,
outputs and
outcomes
   
Input Process Output Outcome Impact

35. More clients get and sustain jobs
Improved work experience
Skills
workshops
Improved basic skills
More
appropriate
behaviour at
work
Greater
confidence
Increased
knowledge of
job
market
Work
placements
Peer mentoring
Reduction in youth unemployment regionally
YOUTH
EMPLOYMENT
PROJECT

36. HOW DO YOU DO IT?
Describe WHAT
change needs to
happen first
Show HOW
changes will be
made by
delivering outputs
Long-term
impact
Necessary
pre-condition
Necessary
pre-condition
Necessary
pre-condition
Necessary
pre-condition
Necessary
pre-condition
Outcomes
must be
achieved
BEFORE long-
term impact
Output Output

37. MONITORING AND EVALUATION FRAMEWORK
3
7
CORNERSTONE OF (SELF) EVALUATION:
Outcomes Outcome
indicators
Information
collection
methods
When and by
whom
How to report
and use
Outputs Output
indicators
Information
collection
methods
When and by
whom
How to report
and use

38. “NCVO CES offers consultancy, training and
publications to the voluntary sector and its
funders.
This increases knowledge and skills in
evaluation, and helps people collect better
data to inform their decision making.
This helps them improve their services
which improves the lives of vulnerable
people at the frontline.”
SUPER QUICK THEORY OF CHANGE

39. EXERCISE:
You have 3 minutes to create your own mini
theory of change, in three sentences:
1. Describe what you do
2. The changes you bring about
3. The long-term, wider change that your
work contributes to
Max 20 words a sentence!
Share it with your neighbour
See if you can help improve each others’

40. FIND OUT MORE
Theory of change:
• Making Connections: Using a theory of change to develop
planning and evaluation
Free download at www.ces-vol.org.uk
• Creating your theory of change: NPC’s practical guide
Free download at www.thinknpc.org
Impact practice:
• Inspiring Impact - http://inspiringimpact.org/
Email: ces@ncvo.org.uk
Tel: 020 7520 3193
Twitter: @CESOnline

41. ROUNDTABLE DISCUSSIONS

42. LUNCH

43. WHAT DIFFERENCE DO YOUR
VOLUNTEERS MAKE?
TOP TIPS ON HOW TO
UNDERTAKE A VOLUNTEERING
IMPACT ASSESSMENT
JOANNA STUART, IVR

44. VOLUNTEERING IMPACT ASSESSMENT
TOOLKIT
Guidance and tools
to help you assess
the impact of
volunteering on:
 Your volunteers
 Your organisation
 Service users
 Wider community

45. PLAN
DOASSESS
REVIEW

46. IDENTIFY WHAT YOU ARE TRYING TO
ACHIEVE/USE YOUR THEORY OF
CHANGE
INVOLVE OTHERS AND WORK
TOGETHER
IDENTIFY CLEARLY WHY YOU WANT
TO ASSESS IMPACT AND WHO IS IT
FOR?
BE REALISTIC AND PROPORTIONATE
PLAN

47. MAKE USE OF AVAILABLE TOOLS AND
FRAMEWORKS
BE PROPORTIONATE – WHAT IS
NECESSARY? WHAT IS MOST
USEFUL?
USE EXISITING INFORMATION AND
EVIDENCE
DO

48. VIAT FRAMEWORK

49. BUILD IN ENOUGH TIME FOR
ANALYSIS AND MAKING SENSE OF
DATA
THINK ABOUT POSITIVES AND
NEGATIVES – BE HONEST AND OPEN
ASSESS

50. BE CREATIVE IN HOW YOU SHARE
AND USE YOUR FINDINGS
USE THE LEARNING – BE WILLING TO
CHANGE YOUR VOLUNTEER
PROGRAMME/PROJECT AND ACT ON
WHAT YOU FIND
REVIEW
FEEDBACK TO THOSE INVOLVED
CELEBRATE!

51. IVR undertakes research, consultancy and
evaluation on volunteering and is part of NCVO.
www.ivr.org.uk
For further information on the toolkit:
www. ncvo.org.uk/practical-
support/publications/volunteering
Next toolkit training will be on July 11th in London
www.ncvo.org.uk/training-and-events
5

52. Impact – Challenges and Tools
Claire Knight – Breast Cancer Care
Rebecca Stewart – Age UK
May 2016

53. What difference can we make?
NNVIA and NCVO members

54. Challenges
 Third Sector purpose is to make social progress
 £43Bn income – means that we can (and do!)
change millions of lives
 Donations, and funding, means scrutiny
 Every project should ask: ‘what difference is being
made ?’
 Impact, and not simply delivery

55. Standardising impact
assessment
 Umbrella bodies are well-placed to drive
standardisation
 Rationale – help the sector make more informed and
visible comparisons
 In order to do more that works

56. So how do we put this
into practice?

57. The power of a peer group

58. Background
• Who got involved?
• What are we trying to achieve?
• Why do we think this is important?
• Why so many dogs?

59. Who got involved?

60. Where did we start?
• Designed a process for capturing our views
• Created a dummy project to test each tool
• Established criteria to measure against
• Created a list of tools to consider
• Invited Sally Cupitt to talk to us

61. Which tools & resources
would we review?
• VITA / VIAT
• VODG / NCF
• Better Impact
• Volunteers Count
• Inspiring impact
• Social Return on Investment
• Balanced scorecard
• Theory of change

62. What did we learn?
• Information & guidance is easily available
• There are plenty of tools to choose from
• It’s important to pick the right tool for the right job
• Plan ahead to ensure the right data is collected
• Beware the common pitfall: measuring outcomes of
projects, instead of impact

63. Next steps
• Complete review of all tools
• Share with all NNVIA members (September 2016)
• Asked NNVIA members if anything missing?
• Decided to invite Sally Cupitt to speak to all NNVIA
members (June 2016)
• Create a “Which Guide” to share

64. QUESTIONS

65. ‘VOLUNTEERING IN
CARE HOMES’
MATTHEW HILL
NVF MAY 2016

66. Volunteering in care homes pilot project
• 3-year Department of Health funded project
• Place volunteers in befriending and activity-
based roles in care homes
• Partnership model between NCVO, Volunteer
Centres (5) and Care Homes (15)
• Over 259 volunteers, almost 10,000 hours
over two years
• Evaluation aimed to draw out good practice
learning and assess impact

67. The context
• Burdens on care homes
• Exceptionally high time pressure
• Many new to volunteering
• One-step removed from evaluation
• Particular user group - especially dementia
• 80% of residents dementia or severe
memory problems
• Capacity to consent
• Has to be face-to-face

68. Changing and flexible pilot
• Not a set ‘intervention’
• Uncertainty around focus of impacts
• Two stage process
• Interim evaluation – explore impacts and
reflect
• Project completion – assess impact
• Further research

69. Social & emotional vs mental & physical
Social & emotional vs mental & physical
• Take a broad approach initially
• Draw on existing frameworks
• VIAT
• ASCOF, Senses and CQC
• Combining can invalidate the tools and
reduce comparability
• Try to explore all types of impact

70. Subjective vs objective
• Practical challenges for objective
measures
• care home burden / non-disclosure
• residents
• Subjective reliable for wellbeing
• Subjective but systematic
• Link with existing evidence
• … but does not satisfy everyone

71. Staff perception of impact
on residents (n=29 of 38)
(%)
Major
positive
Moderate or
minor
positive
No
impact
Major,
moderate or
minor negative
Emotional wellbeing 62 31 3 3
Social wellbeing 59 38 3 0
Mental wellbeing 48 45 7 0
Physical wellbeing 41 38 21 0

72. Return on investment
• Volunteer Investment and Value Audit
• Investment – staff time, training, expenses
• Value – equivalent wage rate per volunteer
hour
• Advantages – very simple, total output, gives
a monetary value, comparability
• Limitations – only measures output,
monetising a non-monetary phenomenon,
adopts market conception of value

73. Volunteer Investment and Value Audit
Quarter Investment Value Return
Oct-Dec 13 £10,388 £0 -£10,388
Jan-Mar 14 £17,021 £2,813 -£14,208
Apr-Jun 14 £15,927 £8,199 -£7,728
Jul-Sept 14 £15,334 £15,558 £224
Oct-Dec 14 £18,203 £16,484 -£1,719
Jan-Mar 15 £17,412 £15,010 -£2,401
Apr-Jun 15 £16,947 £17,834 £887
Jul-Sept 15 £19,488 £23,516 £4,028
Oct-Dec 15 £16,919 £23,980 £7,060

74. Quant vs qual (stats vs stories)
• Not an either or
• … but triangulation of data
• Qualitative case studies and in-depth
interviews allows deep understanding of
impacts
• …and the factors that underpin them
• Involved residents and relatives

75. "If I didn't have the volunteer my morale
would be very low. I'd be shouting at
everybody - all of us in here have some
form of dementia. I ring this bell and
nobody comes and nothing happens and
then I scream at the matron and say you
haven't got enough staff and she gets
very annoyed when I say that" [Resident]

76. “instead of sitting in their rooms…. keep
people thinking the whole time and
keeping the brain alert instead of being a
dodo” [Resident]

77. “The communication has broken down a
little bit.. we seem a little bit out of the
loop…it’s a bit disappointing really. Cos we
felt like we were building a rapport with the
volunteers – it seems to have been a bit of
a breakdown really… We need to go back
down to basics …. and start from scratch”
[Home Manager]

78. Care home cake
Where are the volunteers?

79. Communicating impact
• Reports and presentations
• Video:
• Need to convince people to engage with
research outputs
• Helps give the context for understanding
impact
• Promotional video but underpinned by
evaluation findings…

80. https://www.youtube.co
m/watch?v=alORoCYwK
a0

81. References:
Cabinet Office (2015) Community Life Survey 2014-15, Cabinet Office
Naylor, C., Mundle, C., Weaks, L., & Buck, D. (2013) Volunteering in health and care: Securing a sustainable future,
King’s Fund
Alzheimer’s Society (2014) Statistics. Available at
<http://www.alzheimers.org.uk/site/scripts/documents_info.php?documentID=341>
See details of the project. Available at https://www.ncvo.org.uk/practical-support/volunteering/volunteering-in-care-
homes
Davis Smith, J., Gaskin, K., Ellis, A. & Howlett., S. (2015) Volunteering Impact Assessment Toolkit: A practical guide for
assessing the difference that volunteering makes, Institute for Volunteering Research. Available at
https://www.ncvo.org.uk/component/redshop/1-publications/P78-volunteering-impact-assessment-toolkit
Nolan, M. R., Brown, J., Davies, S., Nolan, J. And Keady, J. (2006) The Senses Framework: improving care for older
people through a relationship-centred approach. Getting Research into Practice (GRiP) Report No 2. Sheffield Hallam
University
Gardiner, C. & Barnes, S. (2016) The impact of volunteer befriending services for older people at the end of life:
Mechanisms supporting wellbeing Progress in Palliative Care
Fujiwara, D., Oroyemi, P. & McKinnon, E. (2013) Wellbeing and civil society Estimating the value of volunteering using
subjective wellbeing data Cabinet Office
Kirkman, E., Sanders, M. & Emanuel, N. (2015) Evaluating Youth Social Action, An Interim Report: Does participating
in social action boost the skills young people need to succeed in adult life? Cabinet Office
Ellis Paine, A., McKay, S. & Moro, D. (2013) Does volunteering improve employability? Evidence from the British
Household Panel Survey Third Sector Research Centre

82. Hostedby:
Impact of volunteering on
wellbeing and mental health
Dr. Daiga Kamerāde, Third Sector
Research Centre, University of
Birmingham
d.kamerade@bham.ac.uk

83. Subjective wellbeing and mental health
More: Diener et al. (2009).

84. Evidence
Systematic reviews and meta
analyses
Randomised controlled trials
(experiments)
Cohort and longitudinal panel
designs
Coho
Cross-sectional surveys
Cro
Qualitative interviewsSee reviews:
(Kamerāde, 2015,
Jenkinson et al.
2013).

85. Evidence summarised
– The impact of regular volunteering on well-being is positive, more so
for a sustained regular volunteering and for people with lower levels
of wellbeing to start with ( Binder, M., & Freytag, A. ,2013).

86. How to measure impact of volunteering on
wellbeing and mental health?
• Subjective wellbeing (see Dolan, P., Layard, R., & Metcalfe, R. 2011) (Scale 0-10):
– Life satisfaction:
• Overall, how satisfied are you with your life nowadays?
• Overall, how satisfied are you with the area you live, relationships , employment
prospects etc
– Happiness:
• Overall, how happy did you feel yesterday?
– Anxiety:
• Overall, how worried did you feel yesterday?
• Overall, how worried are you about your financial situation, environmental problems etc.
– Worthwhileness
• Overall, how worthwhile are the things that you are doing in life? (e.g. Volunteering)
• Mental health (presence/absence of depression)
– The 5-item World Health Organization Well-Being Index (WHO-5) (see Topp, et al. 2015).
• Please indicate for each of the 5 statements which is closest to how you have been feeling over
the past 2 weeks
• Over the past 2 weeks ... I have felt active and vigorous, ... my daily life has been filled with
things that interest me etc (Scale 5 (all the time) to 0 (at no time)

87. A basic experimental design for estimating the effects
of volunteering on wellbeing.
Peoplewhohave
expressedan
interestto
volunteer
Experimental
group
(volunteers)
Control
group
(waitinglist)
Pre-test
Pre-test
Volunteering
Not
volunteering
Post-test
Post-test
are
randomly
assigned
to
=

88. Implications for policy and practice
• Volunteering as a public wellbeing and mental health
intervention?
– Not enough robust evidence
– Robust pilots and their evaluation is needed.
• Claims about volunteering:
– ‘Volunteering may improve your wellbeing and mental health’ [not ‘boosts,
improves, or increases’].
– ‘Nearly 60% of 70 volunteers in this volunteering programme reported
improved wellbeing’.
• Design&Evaluation:
– Lenght, frequency and type of volunteering: any realistic chances and mechanisms for
improving wellbeing?
– If wellbeing and mental health are the key expected outcomes&there is a relatively large
number of volunteers: consider pragmatic randomised controlled trials.
– If improved wellbeing and mental health are not the key expected outcomes& there are
very few volunteers: different research questions and variety of designs.
– Specific wellbeing measurements more informative than global/general ones.

89. More information
• Kamerāde, D. (2015) Third sector impact on human
resources and community: a critical review. TSRC
Working Paper Series No.134. Seventh Framework
Programme (grant agreement 613034), European
Union. Brussels: Third Sector Impact
• Jenkinson, C., Dickens, A., Jones, K., Thompson-Coon,
J., Taylor, R., Rogers, M., et al. (2013). Is volunteering
a public health intervention? A systematic review
and meta-analysis of the health and survival of
volunteers. BMC Public Health, 13(1), 773.

90. References
Binder, M., & Freytag, A. (2013). Volunteering, subjective well-being and
public policy. Journal of Economic Psychology, 34, 97-119.
Diener, E., Oishi, S., & Lucas, R. E. (2009). Subjective Well-Being: The Science
of Happiness and Life Satisfaction. In Oxford handbook of positive
psychology (pp. 187)
http://homepage.psy.utexas.edu/homepage/Class/Psy418/Josephs/Wynne
%20Folder/5-Subjective%20Well-Being.pdf
Dolan, P., Layard, R., & Metcalfe, R. (2011). Measuring subjective well-being
for public policy.http://eprints.lse.ac.uk/35420/1/measuring-subjective-
wellbeing-for-public-policy.pdf
Musick, M. A., & Wilson, J. (2003). Volunteering and Depression: the Role of
Psychological and Social Resources in Different Age Groups. Social Science
and Medicine, 56(2), 259-269.
Topp, C. W., Østergaard, S. D., Søndergaard, S., & Bech, P. (2015). The WHO-5
Well-Being Index: A Systematic Review of the Literature. Psychotherapy
and psychosomatics, 84(3), 167-
176.http://www.karger.com/Article/Pdf/376585

91. QUESTIONS

92. CLOSING REMARKS

93. GET IN TOUCH
Volunteering@ncvo.org.uk
@NCVOvolunteers
https://www.ncvo.org.uk/ncvo-volunteering