The ability to measure the effectiveness of your organization’s training and development programs is critical to ensure your learning strategy is aligned to your desired business outcomes. However, having a system in place to accurately measure the impact of your learning programs can often be a complex challenge — One that organizations struggle with the most.
Join this webinar to discover how to enhance employee performance and prove the value of your learning investments, by implementing simple and effective strategies for measuring the impact of your online training programs.
Watch the presentation to learn:
- Common issues and challenges with measuring learning outcomes
- How to align learning outcomes to business objectives
- Best practices for evaluating the effectiveness of your training programs
- How to analyze and interpret learning data to understand the impact of individual learning, and performance
- How to leverage these insights to improve training programs, and enable the organization to make better informed decisions about learning
Evaluate the Effectiveness of Your Online Learning & Training Programs
1. PRESENTER NAME
Presenter Job Title Here
Evaluating the Effectiveness of Your
Online Learning & Training Programs
Presented with Paula Yunker of
Limestone Learning
9. Why is Evaluation Important?
Training should have a purpose:
• To inform
• To teach
• To improve change
Evaluation will identify if learning
has occurred, and if improvements
have been made
9
10. Aligning Learning to Organization Goals
• For training to be valued by leaders
and stakeholders, all training efforts
need to focus on the organizational
goals
• Consider what your stakeholders are
expecting, and the impact on
business training needs to have to
meet expectation
10
Training
Business Goals
11. Strategies for Aligning Learning
11
• Step 1: Identify the business
need
• Step 2: Create the learning
plan
• Step 3: Measure the business
results
12. 12
Ask your key stakeholders and business
partners:
• What are the desired results?
• What are the goals & outcomes?
• What needs to change to meet these
goals & business outcomes?
Step 1: Identify the Business Need
13. 13
• What KPI’s should change?
• What needs to be learned to make those
changes happen?
• What will success look like to you?
Step 1: Identify the Business Need (cont.)
14. 14
Develop a learning and evaluation plan to
meet the learning requirements based on
your answers to Step 1:
• What needs to be learned to achieve
business outcomes?
• What required KPI’s are needed?
• What learning experiences and practice
are needed to become proficient?
Step 2: Create the Learning Plan
15. 15
• When and how will these learning
experiences and practices be best
accomplished?
• How and when will the learning be
measured?
• How will learning be applied to the job,
and how will that application be
reinforced on the job?
Step 2: Create the Learning Plan (cont.)
16. 16
To identify how learning aligns to business outcomes, and
meets stakeholder/business partners needs:
• What changes in business results & key performance
indicators have been realized from the training program?
Step 3: Measure the Business Results
17. 17
• For every dollar invested in training, how many dollars
does the organization get in return? Is the ROI significant
enough to justify training?
• What key intangible benefits (soft data) have been
gained?
Step 3: Measure the Business Results
(cont.)
18. 18
Where do you need to be with data
& analytics, and how do you get
there?
Take a BIG PICTURE view:
Stakeholders want to know if
programs have helped make money,
save money, or avoid costs.
This brings us to … DATA
19. Start by collecting data easy to obtain
19
• System logins by department – are
there some departments or
organization groups that login
significantly less than others?
• Course completions by department –
are there any learners not completing
courses?
• Pre and post test and quiz results
20. Now consider the tougher questions..
20
How can additional data be helpful?
Calculating Success Measures (using sales training as an
example):
1. Measure for each sales
rep, if possible
2. Isolate variables that
effect revenue
3. Corroborate pre-test, test,
and retention quizzes
4. If knowledge is retained,
proceed.
5. Wait 3 months to
compare sales
6. Isolate data for considerations
of seasonality, product, etc.
21. Let’s step back – Typical Challenges
21
• Lack of buy-in from the top
• No ownership of the project
• Inability, or unwillingness to
measure and track
evaluation on the job
• Lack of reinforcement on
the job
22. Overcome the Challenges:
Implementation Strategies
22
What’s needed to gather & measure
data, and evaluate the results?
• Expertise
• Resources
• Organizational Support
23. Implementation Strategies
23
Evaluation tools that support and measure learning on the job
• Create resources that reinforce learning (job aids, reference
handouts, etc.)
• Create follow-up questions
• Ask similar questions to managers and supervisors, internal
clients, and customers to get a fuller picture
24. Implementation Strategies
24
Learning Action Plan
• A contract between the learner and manager, that provides
structure to assist managers and employees to put learning
into practice
• Gives learners the opportunity to reflect on their learning
experience
• Signed by both parties, and provides accountability
25. Poll: Does your organization use some
form of Learning Action Plan?
25
• No
• Yes
• I’m not sure
26. Overcome your Challenges
26
Before Learning:
• Create pre-evaluation that is
directly aligned to learning
content, and the learning
objectives.
• Examples: self-assessment
surveys, written pre-tests,
blog/wiki questions, reading
assignments
27. Overcome the Challenges:
Implementation Strategies
27
During Learning:
• Design formative evaluation tools, and place in strategic spots
throughout content.
• Examples: team exercises, brainstorming, flipchart discussions,
learning games, reflective exercises
After Learning:
• Design an end of course summative evaluation to evaluate how
successful each learner is at meeting learning objectives
28. Evaluation Best Practices & Tips
28
• Make sure you’re measuring learner satisfaction AND
engagement/relevance to the job
• Incorporate statements/questions that are learner centric
• Consider using a 7, 9, or 11 point rating scale to provide a
richer level of feedback
• Don’t jump to conclusions!
29. Recommended Resources
29
• Evaluation websites
• Books/reference material
• Kirkpatrick Evaluation LinkedIn
group
• Training and certification
• Industry and best practice
research sites
• Limestone Learning blog
31. Follow-up
31
• You will receive the
webinar recording & slides
• Post-webinar satisfaction
survey
• Further questions – don’t
hesitate to contact us.
I’d like to introduce you to our presenter - Paula Yunker. Paula is the Managing Partner and Instructional Designer at Limestone Learning. She has taken time out of her busy schedule to be here with us today, so thank you Paula!
I’m Nimritta, I’m a marketing coordinator here at Lambda Solutions, and I will be co-presenting, as well as managing any questions you might have throughout the presentation.
About Limestone
About Lambda
A little bit about Lambda Solutions:
We are LMS experts and a complete solutions provider – we offer everything from the technology, to training and services.
Beyond the technology, our solutions help you see when, and how, eLearning is achieving goals. We’ve developed a reporting and analytics solution to provide deeper insight into your learners, and learning programs.
We have 12 years of experience, over 600 customer implementations, and 1 million active user accounts across most industry sectors.
These are just some of our amazing industry leading clients
Measuring the value to an organization is a complex but important component of any training course or program.
Training should have a purpose – to inform, to teach, to improve and change.
Introducing evaluation and tying it to performance will identify if learning has occurred during and after training and if job performance improvements have been realized.
Measuring improvements in job performance will also provide valuable data for measuring cost-benefits to the organization.
A key requirement for any organization is to ensure training and learning aligns with organizational goals
· In best practice companies, all training programs cascade down from the overall strategic goals.
· No programs are developed and implemented unless they produce results that are identified as critical to the organization.
· There should be explicit alignment between training programs, learning objectives, and business outcomes – the high level organizational initiatives.
· For training to be valued by organizational leaders, all training efforts need to focus on the organizational goals.
· For each training initiative, begin by considering what your stakeholder and business partner/business unit expectations are and what impact on the business you want the training to have.
· Stakeholders and business partners can provide you with valuable information about what business results are needed (“What will success look like to them?”), why they are important, how they align with organizational goals, and what business measurements (if any) currently exist in their departments or business areas.
· The information can then be used when planning and designing new training initiatives.
How do you ensure training aligns with business outcomes?
Step 1 is to identify the business need
To identify the business need, ask your stakeholders and business partners the following questions:
o What desired results from an organizational perspective do you want to realize from training?
o What are the goals and business outcomes of the organization?
o What needs to change within the organization/ business unit(s) to meet those goals and business outcomes?
What key performance indicators (or KPIs) should change?
o Some examples are sales forecasts, customer satisfaction, employee turnover, operational costs, safety performance and inventory.
o What needs to be learned to make those changes happen?
o What will success look like to you?
Step 2 is to Create the Learning Plan
Once you understand the business need, develop a training and evaluation plan to meet the learning requirements associated with that need based on the answers to the following questions:
o What will need to be learned to achieve the business outcomes?
o What required knowledge, skills and attitude (KPAs) are needed?
o What type of learning experiences and practice are needed to become proficient?
When and how will these learning experiences and practice be best accomplished?
o How and when will the learning be measured?
o How will learning be applied to the job and how will that application be reinforced on the job?
Step 3 is to Measure the Business Results
Many organizations don’t complete this third step.
· If you start with the company's business outcomes, you should also end with the company's business outcomes and how the programs you design and deliver affect them.
· To identify how learning aligns to business outcomes and meets the needs of stakeholders and business partners, you should find the answers to the following questions:
o What changes in business results and key performance indicators have been realized from the learning program?
For every dollar invested in training how many dollars does the organization get in return? Is the ROI significant enough to justify training?
o What key intangible benefits (soft data) have been gained?
This brings us to data. The data you collect and analyze will begin to paint a picture of where you need to improve, highlight data-need gaps, and prove indicators on how improvements can be made.
The important thing to remember here is Take a Big Picture View – Your data should provide insight into how company wide learning/training programs performing? Stakeholders want to know if programs have helped make money, save money, or avoid costs.
First, Make money – This can be through sales training, which can come in the form of product knowledge courses, or sales tactic training which help sales improve performance via soft skills.
Save money – This can be process driven training that helps employees complete tasks faster, or provide strategies to cut out inefficiencies. Eg. Manufacturing company that educates workers on how they can save material and machinery depreciation costs by wasting less raw material, and taking better care of equipment
Avoid costs - Falls under compliance training, such as workplace harassment training, or health & safety training. There are hefty regulatory fines that can come down on any organization if they aren’t meeting compliance standards, especially for those in the healthcare sector.
You can start by collecting the data that is easy to obtain:
For example, system logins by department - are there some departments or organization groups that login significantly less than others?
- Course completions by department - are there any learners not completing courses?
- Pre and post test/quiz results by department
This will provide data to see if training is actually being used. If it is, you can drill deeper into the effectiveness of training in various departments.
Now consider the tougher questions, and how additional data might be helpful. For this deeper data, you will need to work closer with departmental managers to understand how training has impacted the bottom line.
How do we calculate success measures? Sales training is one way to understand how training has impacted revenue, so we are going to use that as an example.
Measure for each sales rep if possible
Isolate variables that also affect revenues
Seasonality, increase in lead generation, new talent, product feature changes.
Corroborate pre-test, test, and retention quizzes
If knowledge is retained, proceed
Wait 3 months to compare sales
You can isolate data for considerations of seasonality, product, or feature changes, new talent on the workforce, etc.
This general structure can be used to determine how training focused on saving money and avoiding costs actually contributes to the bottom line. To do this, you will have to isolate any variables that have any impact on sales, other than training specifically.
This not only helps prove effectiveness of the course, but the content of course - “I designed a course to do this, this proves that it is doing that.”
Lack of buy-in from the top/ commitment from appropriate stakeholders.
o Effective evaluation requires commitment and resources to get it established.
o The necessity to track data and information beyond the ‘training room’ environment means that employees across an organization must understand the importance and value of measuring different kinds of results.
o Without the voice of committed leadership from the top of an organization, as well as the action to support the efforts, it’s extremely difficult to be successful.
o Leadership must be ‘sold’ on the benefits the organization will reap from evaluation in order to get their commitment to make it happen.
· Ownership
o Who’s going to be responsible for creating, administering, measuring and gathering the evaluation data?
o The ability to measure and track performance and the value it brings to an organization are both factors that reach beyond the boundaries of a Learning and Development team or team member.
o Often there seems to be no clear ownership of the process.
· Unable/unwilling to accurately measure and track evaluation on-the-job
o Supervisors and managers often lack the time, resources, commitment and knowledge to measure and track employee performance on the job.
o However their participation is crucial for a successful evaluation program.
· Lack of reinforcement on the job
o Robert O. Brinkerhoff, author and an internationally recognized expert in evaluation and training effectiveness, shows that only 15 percent of what is learned during training will be applied on the job if it is not reinforced and monitored. Creating a post-training support structure as part of the initial training design and development process increases the likelihood that resources will produce a measurable increase in performance.
o Providing excellent training does not lead to significant transfer of learning to behaviour and subsequent results without a good deal of deliberate and consistent reinforcement back on the job.
Substantial levels of investment, expertise, resources and organizational support are required to gather and measure data and evaluate results
· It’s important to first identify which training course or program has the greatest impact to the organization and how important the need is to demonstrate the value of learning.
· Involve stakeholders and business partners in this decision making process.
· One very important decision to make is who will be responsible for gathering and analysing data and measuring the impact of learning on the organization.
· Hard and soft data gathering should begin before the training event occurs to create a baseline.
Think of the learning event as an ongoing process rather than as a one-time event. What do the learners need not only during training but once they’re back on the job.
· Build your post-training activities during your program design.
· The evaluation tools you design should answer the question “Is training working?” rather than “Did training work?”
· Involve managers before you begin to design your evaluation
· Find out what effective job performance means to them. Explain how they can effectively support their employees back on the job and identify what tools and support the managers will need.
· Then write your learning objectives to align with what the learner will be able to do after training…”At the end of the training session, learners will be able to effectively facilitate a project kick-off meeting.” Or “At the end of the training session, the learners will accurately enter a customer record into the client registration system.”
· There are a wide variety of level 3 evaluation tools that can be used to support and measure learning on the job.
o Create job aids, reference handouts, and informal video clips of processes as reminders – whatever is needed to reinforce learning.
o Create follow-up questions that the learners answer. This can be an efficient and effective way to transition from level 2 into level 3 (and potentially level 4) evaluation.
o Learners are asked the questions individually or in group interviews or in an online survey with learners approximately three months post training. Make the questions learner centric.
o To more effectively measure changes in behaviour on the job, ask similar questions to the learners’ managers, supervisors, internal clients, customers and business partners to get a fuller picture.
o The other one is a Learning Action Plan which is used prior to training, at the end of training and back on the job after training.
o It’s a contract between the learner and their supervisor or manager that details what will be learned, what has been learned, how the learning will be implemented and used on-the-job and what actions the manager can take to support their employee back on-the-job.
o Learners are also given the opportunity to reflect on their learning experience.
o The Learning Action Plan is signed by both parties and provides accountability.
· The learning action plan also provides the structure to assist managers and employees to put learning into practice, immediately reinforcing new knowledge and skills post-training and improving the likelihood of training being successfully transferred on the job.
· Learning Action Plans are common in best practice organizations.
Before learning:
· Create pre-evaluation that is directly aligns to learning content and the learning objectives.
· Some examples of pre-evaluation are self-assessment surveys (before and after), written pre-tests, blog or wiki questions and reading assignments with questions.
During learning:
· Design a variety of formative evaluation tools and place in strategic spots throughout your content. There should be at least one formative evaluation per lesson since each lesson should have at least one learning objective.
· Some examples of formative evaluation tools are team exercises, brainstorming, flipchart discussions, informal surveys, learning games, reflective exercises, reflection questions to answer, etc.
After learning:
· Design an end of course summative evaluation to evaluate how successful each learner meets the learning objectives.
· Summative evaluations are completed individually and typically marked using a checklist or rating scale.
· A checklist measures quantity – how many questions the learner correctly answered or what tasks they correctly completed
· A rating scale measures quantity AND quality – not only to measure if the learner can achieve the task/activity but how well can they do it.
· Again, there should be at least one question for each objective in the course.
Measure not only learner satisfaction but their level of engagement and relevance to the job.
· Focus less on the room comfort and whether they were happy with the muffins provided and more on the learning experience.
· The goal is to incorporate statements or questions that focus the learner on higher levels of evaluation and get them thinking about how the new learning will benefit them and the organization after training has occurred.
· For example - make all evaluation statements learner-centered. For example, rather than “The facilitator provided debrief activities for participants to demonstrate their learning”, instead use “The debrief activities helped me to effectively practice what I learned”.
· For example, the content was relevant to me and to my job or I estimate that I will apply the following percent of the knowledge/skills learned from this training directly to my job. Then provide a % scale from 0% to 100% in increments of 10.
· Consider using a 7, 9, or 11 point rating scale to provide a richer level of feedback. Only label each end of the rating scale, rather than labeling each number on the scale (e.g., 1=strongly disagree and 7=strongly agree).
· A cautionary note – just because learner ratings were not high on the level 1 evaluation does not mean that they did not learn!
· So use the results carefully and don’t jump to conclusions.
· That’s why it’s important to gather more data at different evaluation levels to get a more rounded and truer picture of the learning and its impact.
Paula has put together some learning evaluation resources for everyone that I will send out in the follow up email.