1. Outcome-Based Management
OCASI Conference, November 5th, 2009
How do you tell your performance story?
Introduction to the concepts and practices of
Performance Measurement
400-45 rue Rideau Street • Ottawa
Ontario • Canada • K1N 5W8
www.bbmd.ca

2. BBMD Consulting
• Leaders in Performance Measurement
• Have worked with hundreds of profit and
not-for-profit organizations and
thousands of individuals in Africa, Asia,
North America and Europe.
• Extensive Performance Measurement
experience with federal, provincial and
municipal governments (currently
supporting the PM efforts of Canadian
and Ontario Governments)
2

3. Objectives for today
1. Understand how government bodies are using Performance
Measurement
2. Share the fundamentals of Outcome-Based Performance
Measurement
3. Discuss why a non-profit agency MUST use this
methodology to remain competitive
4. Explore how this might change the way your program is
delivered
3
Bell, Browne, Molnar & Delicate Consulting

4. Today’s Reality…EVERYONE IS
DOING IT!
www.expectmore.gov
http://www.mnplan.state.mn.us/mm/
www.ratemyteacher.com
http://www.cbc.ca/canada/ottawa/st
ory/2009/10/22/ottawa-ontario-
second-career.html
4
Bell, Browne, Molnar & Delicate Consulting

5. Federal and Provincial Direction
Federal: Provincial
• Internal federal policy commits to having • Results-based plan drives provincial
the WHOLE of the federal government (all allocation and public reporting (see
programs, including internal services) http://www.gov.on.ca/ont/portal/!ut/p/.c
apply the components of a MRRS (see: md/cs/.ce/7_0_A/.s/7_0_252/_s.7_0_A/7
http://www.tbs- _0_252/_l/en?docid=STEL01_105128)
sct.gc.ca/pubs_pol/dcgpubs/mrrsp- • Budget Transparency and Accountability
psgrr/id-cm/id-cm- Act (BTAA), British Columbia
PR_e.asp?printable=True) (www.bcauditor.com/PUBS/2008_09/Rep
• CIC Settlement – the Modernized ort2/Strengthening%20Accountability_WE
Approach B.pdf)
– Outcomes – The modernized approach is • LMDA has resulted in MTCU building one
an outcome based approach common performance framework for all
– Performance Measurement – results in employment services.
terms of outcomes, outputs, and financial
resources will be gathered and monitored
to ensure activities continue to achieve
expected results and link services to
specific settlement outcomes
5
Bell, Browne, Molnar & Delicate Consulting

6. 6
Bell, Browne, Molnar & Delicate Consulting

7. r
D
de
em to P
n
Fu
on ub
st l i c
d
an
ra a
Why Outcome Based
Performance Measurement?
tio nd
d
ar
n
PRIORITY
of fun
Bo
Su d e
to
SETTING
cc r
ce
es
vi
Assists in identifying priorities
s
Ad
(“what gets measured gets
improved”)
RESOURCE
ALIGNMENT
Everyone knows what is Why ALLOCATION
important to management Performance Helps justify the need for
Su p p
(“what gets measured gets Measurement? resources (“what gets
measured gets funded”)
hip
done”)
ort D
Evi d i si o n M a k
ards
ec
e n ce
Stew
ACCOUNTABILITY
to
ollar
Clarity around accountability
obligations (“always ready to report”)
D
ing
Tax
Answerability to the Public and Funder
Bell, Browne, Molnar & Delicate Consulting 7

8. Not just about meeting funding obligations…
Are you complying or competing?
High performance organizations Ministry of Training, Colleges
build their own Performance and University
Measurement Frameworks,
measuring Effectiveness, Efficiency “Employment Ontario”
and Customer Satisfaction across Performance Measurement
funded program boundaries Framework” 2008
– How do you make sure you are as
relevant as possible to your community?
What is your cumulative value add (more
than just the sum of your programs)
– What value can you demonstrate
BEYOND what your funding body is
asking for?
– How can you proactively report your
progress to your city, your MP, your
community, your funders?
Bell, Browne, Molnar & Delicate Consulting 8

9. CASE STUDY: Job Connect
How did MTCU get to be ‘outcome based’
• Longitudinal study over a decade ago showed
that going to an employment agency did not
increase your likelihood of employment
• Program objectives were to serve the ‘hard-to-
serve’ – they were NOT being served
• Something had to change – Introduction of
the Continuous Improvement Performance
Measurement System (CIPMS)
Bell, Browne, Molnar & Delicate Consulting 9

10. Understanding Job Connect:
Congratulations! I just gave you 1.2 million to serve 1000 ‘hard-to-serve’ clients, but
there’s more...
Bell, Browne, Molnar & Delicate Consulting 10

11. 2. Employment Service Funding Decision Matrix
Compliance Overall
Evidence of Reached
with MTCU Service
Organizational Improvement
Contract Quality
Capacity
Standard Targets
MTCU Response
Achieved
Success – Funding approval, Performance Incentive and/or Award
Potential
✔ ✔
Organization is in compliance with contract and program guidelines and has
✔ ✔ achieved or surpassed Overall Service Quality Standard for service delivery
OC review demonstrates ability for sustained and/or improved results
Organization has attained improvement targets identified in last contract
Official Review – Funding Approval with Immediate Compliance
Directive
Organization is NOT in compliance with contract despite achieving or
✘ ✔ ✘ ✘ surpassing Overall Service Quality Standard for service delivery and must
submit an action plan for achieving compliance
OC review demonstrates little ability for sustained or improved results
Organization did not meet improvement targets from last contract.
Directed Improvement - Funding Approval with Compliance and In Year
Improvements to Overall Service Quality Standard
Organization is NOT in compliance with contract/guidelines and must submit an
✘ ✘ ✔ ✘ action plan for achieving compliance
Organization has not achieved Overall Service Quality Standard for service
delivery nor improvement targets from last contract
OC review demonstrates there is capacity and potential for immediate
improvement
Termination – Notice of contract termination, procedures to support
✘ ✘ ✘ ✘ client service transfer
Organization did not achieve in year service improvements to standard and/or
is not in compliance with contract/guidelines
Organization does not demonstrate necessary OC to support improvement
3
11
Bell, Browne, Molnar & Delicate Consulting

12. CIC, Implementing the modernized approach,
October 2009
Bell, Browne, Molnar & Delicate Consulting

13. CIC, Implementing
the modernized
approach, October
2009
Bell, Browne, Molnar & Delicate Consulting

14. An intensive.
2. PERFORMANCE MEASUREMENT
FUNDAMENTALS
Bell, Browne, Molnar & Delicate Consulting 14

15. Performance
Measurement…
is all about answering the following questions:
– Why do we exist?
– What do we want to accomplish / what is our end state?
– How are we going to deliver our Programs?
– How will we know how well we are doing?
– What programs, priorities and initiatives do we need to
undertake to work meaningfully towards our end state?
Bell, Browne, Molnar & Delicate Consulting 15

16. Some Basic Requirements for Success
• Common Glossary and Methodology
• Common understanding and agreement of the
desired ‘end state’
• Establishing the “vital few” performance
measures
• Building a performance report and reporting
strategy
• Formalize a continuous improvement strategy
Bell, Browne, Molnar & Delicate Consulting 16

17. The foundation – the logic model
Short-term Intermediate Long-term Strategic
Inputs Activities Outputs Outcomes Outcomes Outcomes Outcomes
← High Control
Low Influence →
RESULTS
Bell, Browne, Molnar & Delicate Consulting 17

18. Inputs, Activities, Outputs and Outcomes
Definition Examples
Inputs
The human, financial, structural resources,
“what we need
knowledge and information necessary to
to do the work”
perform the activities.
Describe a collection of functions (actions,
jobs, tasks) that consume inputs/resources
Activities
and whose outputs deliver
“what we do”
benefits/impacts to a client or contribute to
a particular outcome/set of outcomes
The direct products and/or services
Outputs
(“usually tangible and concrete”) generated
“what we
through activities without specific
produce”
reference to their ultimate purpose
The benefits or desired states (“not fully
Outcomes
under our influence”) to which the outputs
“why we do it”
of activities contribute
Bell, Browne, Molnar & Delicate Consulting 18

19. Four Types of Outcomes
Categorized according to the degree of influence, as
follows:
1. Immediate Outcomes First-level effects of, or
immediate response to the outputs, e.g., changes in
compliance rates or degree of customer satisfaction.
2. Intermediate Outcomes are the second order of
outcomes, Benefits and changes in behaviour, decisions,
policies and social action attributable to outputs to
demonstrate that program objectives are being met,
e.g., increased employability as a result of a training
program.
Bell, Browne, Molnar & Delicate Consulting 19

20. Four Types of Outcomes
3. Ultimate/Long-term Outcomes The ultimate or long-term
consequences for human, economic, civic or environmental
benefit, to which government policy or legislation contributes,
e.g., life expectancy rates, overall economic performance. Very
broad in nature, requires many departments collaboration and
take some time to achieve.
All of which feed the…
4. Strategic Outcomes are those long term and enduring
organizational and/or government-wide benefits or goals that
the Project, Program or Organization is contributing to.
Bell, Browne, Molnar & Delicate Consulting 20

21. This first, then this, then this, then this….
WHAT CHANGES IN THE WORLD AS A RESULT OF THE WORK YOUR
AGENCY IS DOING?
IMMEDIATE INTERMEDIATE Long Term
OUTPUTS
OUTCOMES OUTCOMES OUTCOMES
Products KNOWLEDGE ATTITUDES BEHAVIOUR EARLY / LATER
EFFECTS
and •Awareness
•Understanding
•Perceptions •Involvement
•Compliance
•Societal change
•Acceptance
Services •Skills •Action
•Socio-economic
benefits
Social Marketing Role
MORE CONTROL LESS CONTROL
(easier attribution) (attribution
difficult)
Bell, Browne, Molnar & Delicate Consulting 21

22. Principles of Good Outcome Statements
1. Outcomes are noun-based desired end states (e.g., “newcomers are
meaningfully employed…” rather than “increase employment
opportunities for newcomers …”)
2. Directional outcome statements (e.g., “Improved/increased labour
market access by Ontarians…”) do not belong in logic models or
outcome statements. Direction is achieved through targets not
outcome statements
3. Avoid self-serving statements (e.g., “World leader in settlement
services”). After all, this is not the end state desired
4. Clear cause and effect linkage to the next level(s) of outcomes
5. SIMPLE: Outcome described in a way that helps Canadians relate to
the desired state that is being described
Bell, Browne, Molnar & Delicate Consulting 22

23. Exercise: Outcomes language
exercise
Small group work
• Examine the bulleted list and reach consensus on which items
are immediate, intermediate, long term outcomes
Bell, Browne, Molnar & Delicate Consulting
4. PM Concepts 23

24. Outcome Exercise
Examine the list and reach consensus on which items are immediate,
intermediate, OR long term outcomes Imm / Inter / LT
1. Newcomers and their families are 8. Canadians provide a welcoming
healthy, safe and secure. community to facilitate the full
2. Our settlement agency is recognized and participation of newcomers into
consulted in settlement-related policy Canadian society
development. 9. Newcomers access affordable childcare.
3. Communities know, understand and 10. Certainty over ownership and use of
appreciate the social, cultural and land and resources by First Nations and
economic benefits that newcomers Inuit communities.
bring. 11. Newcomers and their family actively
4. A skilled, adaptable and inclusive labour works towards achievement of goals
force and efficient labour market. 12. Newcomers are aware of and access
5. Preservation of the Environment. settlement programming (e.g.: labour
6. Women and Men have equal treatment market language training with a bridge-
under the law. to-work component)
7. Newcomers have the job search skills
required.
Bell, Browne, Molnar & Delicate Consulting
4. PM Concepts 24

25. Settlement Logic Model
Direct Outcomes Shared Outcomes Ultimate Outcomes
Bell, Browne, Molnar & Delicate Consulting 25

26. 1. Outcomes
Considering these outcomes, what SHOULD we be measuring?
Measured,
Outcomes
(ok to stick post-it notes to the relevant column)
Managed
Direct Outcomes Shared Outcomes Ultimate Outcomes
Bell, Browne, Molnar & Delicate Consulting

27. Indicators and Targets
• Indicators ( The “How”)
– A statistic or parameter that, tracked over time, provides information
on trends in the condition of a phenomenon (e.g., percentage of
clients that are satisfied)
• Targets (The “How Much”)
– Specific performance goals tied to indicators against which actual
performance will be compared (e.g., 90% client satisfaction rating)
• Standards
– Standards imply a service level to which an organization is prepared to
commit, either internally, externally or both.
Bell, Browne, Molnar & Delicate Consulting 27

28. Examples - Indicators
• % of newcomers who are aware of the skills and expectations necessary to join the
Canadian labour force
• % of newcomers who access to labour market language training
• % of clients who report that they have the language ability and skills needed to
participate socially, culturally and economically in Canada
• % of clients who report that they feel connected to the broader community and
social networks
• % of program participants who claim to create a welcoming community for
newcomers
• % employed for > than 3 months
• % satisfied with service
Bell, Browne, Molnar & Delicate Consulting 28

29. Examples - Targets
• 80% of newcomers aware of the skills and
expectations necessary to join the Canadian labour
force
• 90% of newcomers have access to labour market
language training
• 64% employment retained for > 3 months
• Full integration of newcomers as participating
members of Canadian society (economic, education,
employment, etc.) within 10 years of arrival.
Bell, Browne, Molnar & Delicate Consulting 29

30. CASE STUDY –
Employment Ontario
Achieving the standard of overall service quality is
key to stable and ongoing funding
For illustration purposes only!! Provincial Service Quality Standard
MINIMUM
PROVINCIAL SQS MAXIMUM
DIMENSION MEASURE STANDARD WEIGHT VALUE VALUE
Customer Service 1. Customer Satisfaction 85% 15% 1.28 1.5
(40%) (participant/employer)
2. Service Coordination 30% 25% 0.75 2.5
Effectiveness 3. Employed/Career Path 69% 25% 1.73 2.5
(50%)
4. Training/Education 10% 10% 0.10 1.0
5. Suitability (Participant 25% 15% 0.38 1.5
Profile)
Efficiency (10%) 6. Intake in assisted 90% 5% 0.45 0.5
service
7. Workshop 90% 5% 0.45 0.5
Activities/Information
Sessions
Service Quality Standard 10.0
Customer Satisfaction (0.85 x 1.5) = 1.28
Service Coordination (0.30 x 2.5) = .75
Employed/Career Path (0.69 x 2.5) = 1.73
Training/Education (0.10 x 1.0) = 0.10
Suitability (0.25 x 1.5) = 0.38
Intake in Assisted Service (.90 x .5) = 0.45
Workshops (.90 x .5) = 0.45
Overall Service Quality Standard 5.14
30
Bell, Browne, Molnar & Delicate Consulting

31. How to Determine “the Critical
Few” Indicators
• Credible – provide a credible, independent view
• Specific - eliminate ambiguity, show relevance to the expected outcome
• Linked - clear (cause and effect) linkages to other indicators
• Reliable - scientifically and statistically sound, provide an appropriate
degree of accuracy. Measure the same thing across time (allowing
comparisons) and for different groups/regions
• Available – data are easily accessible or there is a low level of effort to
collect and analyze
• Cost-effective – the costs for collection are aligned with the overall utility
of the indicators
• Understandable – data can be easily grasped by various audiences
Bell, Browne, Molnar & Delicate Consulting 31

32. Performance Indicator Assessment Grid
Priority
Degree of importance/relevance in terms of measuring efficiency (“doing things well”) and/or effectiveness
(“doing the right things/having an impact”)
High 3 It is essential in order to measure efficiency and/or effectiveness
Medium 2 It is useful in measuring efficiency and/or effectiveness
Low 1 It is of little use in measuring efficiency and/or effectiveness
Data Accessibility
The degree to which data are readily available either manually and/or through an automated system
High 3 Indicator data are immediately available
Medium 2 The appropriate data are not immediately available through an automated or manual system
but could be gathered relatively easily
Low 1 The data to support this indicator are not currently available or accessible and will be difficult
to access
Level of Effort
An estimate of the level of effort required to report reliably against the indicator on a regular basis
Low 3 Minimal effort to gather, analyze, interpret and report reliably on the indicator
Medium 2 It requires a moderate degree of effort to reliably report on the indicator
High 1 It is very resource intensive to reliably report on the indicator
Bell, Browne, Molnar & Delicate Consulting 32

33. Type here
Discussion.
WHY MUST A NON-PROFIT USE THIS
METHODOLOGY TO REMAIN COMPETITIVE?
Bell, Browne, Molnar & Delicate Consulting 33

34. Type here
Discussion.
HOW COULD THIS CHANGE THE WAY YOUR
SERVICE IS DELIVERED?
Bell, Browne, Molnar & Delicate Consulting 34

35. SARAH DELICATE, BBMD
(905) 922-9726 (cell), sarah.delicate@bbmd.ca
Bell, Browne, Molnar & Delicate Consulting 35