1. Business Impact
The Long-Term
Value of Using
Metrics That
Matter®
KnowledgeAdvisors
17th March 2011

2. Objectives
• Discuss the value proposition for
measuring learning.
• Describe the long-term value of using
Metrics That Matter®
14-Mar-12 © 2009, KnowledgeAdvisors 2

3. Audience Poll
 What is your role within your organization?
 A) L&D professional
 B) L&D Leader
 C) HR professional
 D) Business leader
 E) Other
14-Mar-12 © 2009, KnowledgeAdvisors 3

4. Audience Poll
 Are you currently a user of Metrics That
Matter®
 A) Yes
 B) No
14-Mar-12 © 2009, KnowledgeAdvisors 4

5. Agenda
 Archetypal L&D Organization
 Why measure learning effectiveness?
 How do KnowledgeAdvisors’ clients
measure learning impact?
 What are the long-term benefits?
14-Mar-12 © 2009, KnowledgeAdvisors 5

6. Archetypal L&D Situation
―I can’t improve ―Don’t tell me it will
it if I can’t be a cold day in ―I don’t want to
measure it.‖ Hell before we get leave you out in
a measurement the cold on this
strategy.‖ one, so leverage
your whole team.‖
CLO
L&D
Director
―It’s a very cold winter day outside. Unless we
develop a measurement strategy, we’re going to
be out in it looking for new jobs.‖
L&D Manager
14-Mar-12 © 2009, KnowledgeAdvisors 6

7. We Know What Leaders Want…
N = 101
14-Mar-12 © 2010, KnowledgeAdvisors 7

8. Why Measure Learning?
L&D is an integral part of the business
designed to close gaps and help the
workforce achieve business goals.
14-Mar-12 © 2009, KnowledgeAdvisors 8

9. How Do Clients Measure Learning?
 Learning should be measured in a way
that aligns with business objectives.
 Usually this means demonstrating:
 Satisfaction with learning
 Knowledge and skill acquisition
 Application of learning on the job
 Demonstration of learning’s impact on the
business (e.g., sales, quality, customer sat.)
14-Mar-12 © 2009, KnowledgeAdvisors 9

10. Most Common Methodologies
 Kirkpatrick’s 4 Levels of Evaluation
 Phillip’s ROI Methodology
 Bersin’s Learning Impact Analysis
 Brinkerhoff’s Success Case Method
 All of these individuals are or have been
on our Board of Directors and have
influenced the development of Metrics
That Matter®
14-Mar-12 © 2009, KnowledgeAdvisors 10

11. The Goal of Training Evaluation
 Demonstrate cause and effect
Improved
Performance Training Performance
 Determine gaps or deficiencies in training
that require continuous improvement
March 14, 2012 11 © 2009, KnowledgeAdvisors

12. KnowledgeAdvisors: What We Do
Experts in
metrics that
matter KnowledgeAdvisors is a human
capital analytics firm. We are the
world’s largest provider of learning
and talent measurement solutions.
March 14, 2012 © 2009, KnowledgeAdvisors 12

13. Integrate Measurement with Management
March 14, 2012 © 2009, KnowledgeAdvisors 13

14. Showing Business Impact
High Requires actual results
Investment data and advanced
analysis techniques
Typically Actual Causal
Results Modeling
employs Correlations
surveys Business
Impact
Templates
Advanced
Business
Human Results
Capital Evaluation
Approach Determine the
Smart appropriate analysis,
Sheets
then select method
Low
Investment
Simple Complex

15. Business Case for L&D Measurement: A Financial Link
Shareholder
Returns of
companies
investing in L&D
measurement
have 17% higher
returns than the
S&P 500
Source: McBassi Investments, 2009
The is the cause and effect relationship we hope to demonstrate!
March 14, 2012 15 © 2009, KnowledgeAdvisors

16. Long-term Use of Metrics That Matter®
 Investigated long-term clients that have used MTM
consistently for 5+ years
 Gathered vast data sets and focused on key metrics:
 Knowledge and skill acquisition (L2)
 I will be able to apply training to the job (L3)
 Training will improve my job performance (L4)
 Training was a worthwhile investment (L5)
 An average of all 4 metrics
 We expected that scores would rise with time because
organizations monitor performance, cull under-
performing courses, and continuously improve their
curricula.
14-Mar-12 © 2009, KnowledgeAdvisors 16

17. Initial Sample: Telecom Company
We examined the metrics for one company first to see if our
hypothesis seemed tenable. Here are our results.

18. Initial Sample: Telecom Company
Each year large numbers of evaluations were collected.

19. Initial Sample: Telecom Company
We computed the performance increase as a percentage:
(2010 value – 2006 value) / (2006 value)

20. Initial Sample: Telecom Company
Some data were not collected every year, but the trend upward
was evident even with only three years of data.

21. Initial Sample: Telecom Company
The combination of all metrics provides a good summary measure.
Despite the variability across the four metrics, the average
performance still improved 14.9%.

22. Initial Sample: Summary
When all 5 metrics are graphed, it is clear that performance rises over time.
The number of evaluations is also overwhelmingly large lending undeniable
reliability to the results. The average increase is also substantial.

23. Case Study vs. Trend
 The Telecom company is only one example.
 Does the hypothesis hold true for other
companies or is the telecom example an
anomaly?

24. Companies with 5 Years of Data
Four organizations contributed 5 years of data:
Two insurance companies, a hospital and a telecom company

25. Companies with 6 Years of Data
Six organizations contributed 6 years of data:
An office solutions company, an Army Base, a government
defense agency, a pharmaceutical company, a US-based
accounting firm, and a UK-based accounting firm

26. Other companies?
 The analysis will continue soon with other
companies:
 Small and medium businesses
 Learning Providers
 We expect the results will be similar.
14-Mar-12 © 2009, KnowledgeAdvisors 26

27. Conclusion
 Measurement leads to performance improvement
for L&D departments
 The size of the improvement ranges from 3 – 5%
on average but can be as large as 14.9%
 Revisit Lauri Bassi’s work—investments in HC
lead to business improvement—outpacing the
S&P 500 by 17%
 MTM helps make learning organizations more
effective through measurement.