1. Managing Sound In-house
Evaluations
Elga Nikolova
Copyright e-NOVA Consulting, 2012.This work is the intellectual property of the author. Permissions
granted for this material to be shared for non-commercial, educational purposes, provided that this
copyright statement appears on the reproduced materials and notice is given that the copying is by
permission of the author. To disseminate otherwise or to republish requires written permission from the
author
OCASI Professional Managing Sound In-house Evaluations e-
1
Development Conference, 2012 NOVA Consulting

2. Our perspective today
We will focus on
 Program evaluation
 Goal-oriented evaluation
 Practices that work, rather than theory
 Managing vs. doing
OCASI Professional Managing Sound In-house Evaluations
2
Development Conference, 2012 e-NOVA Consulting

3. Why evaluation?
 To identify gaps
 To find out if a program is working well
 To make sure clients are satisfied
 To measure the impact of programs and services
 To improve programs and services
 To ensure that programs contribute to the organization’s
vision and goals
 To justify requests for funding
 To meet funding requirements
 To assist with planning
 To find the value we are getting for money spent on a
program
OCASI Professional Managing Sound In-house Evaluations
3
Development Conference, 2012 e-NOVA Consulting

4. Evaluation
IS A DECISION-MAKING TOOL
OCASI Professional Managing Sound In-house Evaluations e-
4
Development Conference, 2012 NOVA Consulting

5. The manager’s role in the evaluation
Capacity
Possibility
Resources Ac t i o n
Program set up
(program Tools and
evaluability) processes
Review and
analysis of data
Decision-making
OCASI Professional Managing Sound In-house Evaluations e-
5
Development Conference, 2012 NOVA Consulting

6. BUILDING CAPACITY FOR
EVALUATION
OCASI Professional Managing Sound In-house Evaluations e-
6
Development Conference, 2012 NOVA Consulting

7. Capacity building blocks
 Champion
 Competence
 Cost
 Resources
 Common language
 Responsibilities
 Schedule
OCASI Professional Managing Sound In-house Evaluations
7
Development Conference, 2012 e-NOVA Consulting

8. Five things to keep in mind about costs
 You will have to invest time even if you hire an
external evaluator
 Make time for your staff to participate
 Technology and good process may take time
to establish but they save time down the road
 Budget time to review and analyze the
collected data
 Beware of the hidden costs
OCASI Professional Managing Sound In-house Evaluations
8
Development Conference, 2012 e-NOVA Consulting

9. Three tips about evaluation resources
• Invest in basic technology: SurveyMonkey,
Skype
• Develop staff competence for evaluation
• Use external consultants smartly
OCASI Professional Managing Sound In-house Evaluations
9
Development Conference, 2012 e-NOVA Consulting

10. The importance of common language
• What do these terms mean?
Deliverable
OCASI Professional Managing Sound In-house Evaluations
10
Development Conference, 2012 e-NOVA Consulting

11. The need for common evaluation
language
 What terms do we use?
 What terms do our funders and stakeholders
use?
 When we say the same word, do we mean the
same thing?
OCASI Professional Managing Sound In-house Evaluations
11
Development Conference, 2012 e-NOVA Consulting

12. MAKING EVALUATION POSSIBLE
PROGRAM DESIGN AND PROGRAM EVALUABILITY
OCASI Professional Managing Sound In-house Evaluations e-
12
Development Conference, 2012 NOVA Consulting

13. Fundamental requirements for evaluation
 The target population is clearly defined
 The program has objectives, specified
deliverables and outcomes
 There is a logical connection between needs,
objectives, actions, deliverables and
outcomes: theory of change and program
logic model
OCASI Professional Managing Sound In-house Evaluations
13
Development Conference, 2012 e-NOVA Consulting

14. If you do not know where you are going,
how will you know if you are getting there?
OCASI Professional Managing Sound In-house Evaluations
14
Development Conference, 2012 e-NOVA Consulting

15. Theory of change
The Turkey Dinner Project
Purpose: to grow our friendship
???
Objective: serve a great Thanksgiving dinner
to make friends feel happy and appreciated
OCASI Professional Managing Sound In-house Evaluations
15
Development Conference, 2012 e-NOVA Consulting

16. The Turkey Dinner Logic Model
Why am I doing this? (Purpose and Objective)
To make friends feel happy and appreciated
To treat my friends to a nice, memorable Thanksgiving Dinner
Inputs Outputs Outcomes
One turkey
Activities Product (what?) Process /quality So What?
(what?) One roast turkey outcomes (how?)
Cooking
Stuff and roast Table arranged for 16
ingredients Turkey juicy and
turkey Short-term Long-term
tender
Cooking skills Lay table
Music appropriate Friends enjoy
Plan seating for the moment.
Participation (Who?) dinner Stronger
Select DVDs 16 guests attend Everyone involved
Seasonal Friends feel friendships
decorations Serve dinner dinner in conversation happy during
Maintain Guests stuff holidays
Nice music conversation themselves on Friends feel
turkey appreciated
Assumptions: good food makes friends happy, all External Factors: Weather, friends’ availability, friends’
friends like music, all friends like turkey, host will have mood upon arrival, guests’ like/dislike for each other
enough time to prepare
OCASI Professional Managing Sound In-house Evaluations e-
16
Development Conference, 2012 NOVA Consulting

17. ACTION
FROM DATA TO KNOWLEDGE TO PRACTICE
OCASI Professional Managing Sound In-house Evaluations e-
17
Development Conference, 2012 NOVA Consulting

18. Five stepping stones from data to
decision-making
 Determine what data to collect, when and
how
 Select a comparison framework
 Have tools and processes in place
 Implement your evaluation
 Once in a while look at the findings
OCASI Professional Managing Sound In-house Evaluations
18
Development Conference, 2012 e-NOVA Consulting

19. Collecting good data
 How do you know that change is happening?
 Attitudes
Knowledge
Behaviour
Action
 Involve clients and staff
 Use observations
 Count
 Keep it small
 Collect data about process and quality
OCASI Professional Managing Sound In-house Evaluations
19
Development Conference, 2012 e-NOVA Consulting

20. Choosing a comparison framework
 There is no evaluation without comparison
 Your choice of comparison
frameworks:
 Baseline data
 Standards set by funders
 Previous experience
 Literature
 Similar programs in the community
 Agree on a comparison framework
from the start of the program
 Your comparison framework could change
OCASI Professional Managing Sound In-house Evaluations
20
Development Conference, 2012 e-NOVA Consulting

21. Tools: SurveyMonkey
PROS CONS
 Stores data  One account, one login
 Provides sample questions  Requires some survey design
 Makes data entry easy knowledge
 Can keep track of separate client  Has a learning curve
groups  Has to be managed
 Makes analysis easy
 Makes presentation easy
 Free for small questionnaires
OCASI Professional Managing Sound In-house Evaluations
21
Development Conference, 2012 e-NOVA Consulting

22. House tools
 Observation templates
 Excel spreadsheets
 Forms
 Process descriptions: how do you store,
analyse and report data?
OCASI Professional Managing Sound In-house Evaluations
22
Development Conference, 2012 e-NOVA Consulting

23. Implement your evaluation
 Evaluation is a project: use common project
management sense
 Align evaluation timing with the program cycle
 Give your participants a chance to speak to a
third party
OCASI Professional Managing Sound In-house Evaluations
23
Development Conference, 2012 e-NOVA Consulting

24. You have paid for it: use it
 Once in a while, look at your findings
 Share success and challenges
 Have evaluation results ready for reporting
and proposal writing times
OCASI Professional Managing Sound In-house Evaluations
24
Development Conference, 2012 e-NOVA Consulting

25. More questions?
Elga Nikolova
elga@e-novaconsulting.com
416 732 7979
OCASI Professional Managing Sound In-house Evaluations
25
Development Conference, 2012 e-NOVA Consulting