Executing a BUSINESS-FOCUSED MULTI-RATER ASSESSMENT TO DRIVE BUSINESS OUTCOMES.A Pharmaceutical company case study

U S I N G M U L T I - R A T E R / 3 6 0
A S S E S S M E N T S T O D R I V E
B U S I N E S S O U T C O M E S
O v e r v i e w
Multi-rater / 360 assessments are used by many organizations
to provide leaders with valuable information about their job
performance from a variety of sources, including managers,
peers, subordinates, and customers. Each of the different
rating sources contributes unique information about the
leader’s performance, thus enhancing the validity of the
performance assessment and helping leaders to more
accurately identify their strengths and opportunities for
development.
By
Shane Douthitt, Ph.D., Executive Consultant
Scott Mondore, Ph.D., Executive Consultant

2 © 2014 Sales Performance International, Inc
E X E C U T I N G A B U S I N E S S - F O C U S E D M U L T I - R A T E R A S S E S S M E N T
F i g u r e 1 . T h e M u l t i - R a t e r A s s e s s m e n t P r o c e s s
SALES PERFORMANCE INTERNATIONAL // WHITE PAPER
Using Multi-Rater/360 Assessments to Drive Business Outcomes
D e f i n e
O b j e c t i v e s
D e v e l o p T o o l
& P r e l a u n c h
D e l i v e r
R e p o r t s &
F e e d b a c k
D e f i n e
C o m p e t e n c i e s
L a u n c h
A s s e s s m e n t
C o n d u c t
S y s t e m a t i c
A n a l y s i s
S t e p 1 : E s t a b l i s h a n d C o m m u n i c a t e O b j e c t i v e s / I n t e n d e d O u t c o m e s
Clearly stated, well-communicated objectives are critical to the effective implementation of any multi-rater assessment
process. In particular, it is important to communicate the business needs being met by the implementation of the multi-rater
assessment, the ways in which individuals will benefit by participating, and the impact the process will ultimately have on the
organization. During this step, the following questions should be answered:
• Which employees will participate in the process?
• How will reports be delivered and feedback facilitated?
• What competencies/behaviors will be assessed?
• What role will the managers of participants play?
• How will raters be selected?
• How will the overall process be communicated?
Key organizational stakeholders should be involved in establishing the objectives of the process and in communicating these
objectives to employees.
S t e p 2 : D e f i n e C o m p e t e n c i e s
The next step is to identify and define the competencies on which participants will be rated. Ideally, these competencies
will come directly from your organization’s validated competency model. The set of competencies selected for inclusion
on the multi-rater assessment should encompass the specific behaviors expected of individuals in their job. Examples of
competencies for sales representatives in the pharmaceutical industry include Product and Disease-State Expertise, Core
Selling Skills, and Managed Care and Reimbursement Expertise.
S t e p 3 : D e v e l o p T o o l a n d P r e - L a u n c h
At this point, it is time to develop the assessment tool (i.e., online assessment). This tool will be used to capture ratings
from multiple sources on the competencies identified in the previous step. During this step, it is also critical to begin
communicating the assessment launch to key stakeholders. Effective communications will help to clarify the purpose of the
assessment, align the initiative with broader organizational strategies, outline project timing, reduce participant anxiety, and
improve response rates.
S t e p 4 : L a u n c h A s s e s s m e n t
It is now time for participating employees to invite their raters, including managers, subordinates, peers, and customers,
to complete the assessment. To improve response rates from customers, employees should clearly communicate the
organization’s ongoing efforts to improve customer service by obtaining customer feedback and acting on their input.
Furthermore, once the assessment process is complete, employees should provide customers with a summary of the
developmental feedback received and an outline of the activities planned to address this feedback.

S t e p 5 : D e l i v e r R e p o r t s a n d F e e d b a c k
Participants should receive detailed feedback reports outlining their strengths and development needs. Reports may include
overall competency scores as well as the participant’s scores on each assessment item. To protect the anonymity of raters
(i.e., peers, subordinates, customers), scores by rater group should only be broken out for groups in which three or more
ratings were received (except for manager and self –ratings which are provided based on a single response). Participants
should use their feedback to identify areas in which growth is needed and build development plans.
S t e p 6 : C o n d u c t S y s t e m i c A n a l y s i s
Finally, the multi-rater assessment results should be analyzed at the organizational level. Using the aggregated results of
individuals across the organization, it is possible to identify the competencies that drive critical business outcomes (e.g.,
sales dollars, market share). This analysis allows organizations to better focus training and development initiatives on the
individual behaviors that contribute to desired business outcomes. This step will be illustrated with a case study.
L I N K I N G Y O U R M U L T I - R A T E R A S S E S S M E N T T O B U S I N E S S O U T C O M E S :
A C A S E S T U D Y
The sales leadership at a small pharmaceutical company wanted to invest in the individual development of its specialty sales
representatives and identify the critical skills needed to gain market share on its primary competitor. The company’s brand
possessed a 23.3% share of the market. This particular market is relatively mature and the organization’s leaders were
looking for a new commercial edge in an extremely competitive market. The following reveals the broader challenges facing
the organization:
• Need to invest in the development of specialty pharmaceutical sales representatives
• Need to identify the critical skills for its sales representatives
• Need to grow market share in a competitive, mature market
• Need better understanding of the ROI of organizational training investments
• Need to drive revenue growth
I m p l e m e n t i n g t h e 3 6 0
Organizational leaders decided to implement a development-focused 360 assessment initiative. The performance ratings
were to be used to identify strengths and development opportunities among individual sales representatives, but the results
were not linked to personnel decisions. This was done to build trust in the process and encourage sales representative
participation. To reinforce this message, individual reports were delivered solely to the sales representative such that they
owned the data and results. However, sales representatives were encouraged to share their results with their manager,
thereby allowing them to leverage the coaching process and development tools made available to them.
A s s e s s i n g S a l e s R e p r e s e n t a t i v e C o m p e t e n c i e s
The multi-rater 360 consisted of 25 skill and behavior-based items representing seven sales representative competencies:
Account Management, Core Selling Skills, Relationship Building, Consultative Selling, Product & Disease-State Expertise,
Evidence-Based Selling®, and Managed-Care Expertise. There was also an open-ended item on which managers and
physicians could comment about how the sales representative could best improve his/her performance. Each sales
representative personally invited individual physicians to complete the short on-line survey (no incentive was offered to
physicians to complete the survey). Physicians were identified by the organization based on territory market share potential.
Managers completed an assessment of the same competencies for each sales representative. Finally, each representative
completed a self-assessment of the same competencies to aid in the development planning process.

E v i d e n c e - B a s e d
S e l l i n g - S k i l l s
1
A s s e s s m e n t P a r t i c i p a t i o n
One hundred and sixteen sales representatives participated in the business-focused 360. A total of 1,266 physicians were
sent a survey invitation and more than 50% responded—a very strong physician response rate. On average, 7.5 responses
were received for each sales representative, including 5.5 physician responses, a manager response, and a
self-assessment.
S a l e s R e p r e s e n t a t i v e s a n d M a n a g e r R e s u l t s
Individual sales representatives were each provided a summary report of their assessment results. Furthermore, managers
and sales representatives attended feedback workshops designed to assist them in understanding the reports, identifying
strengths and development opportunities, building development plans, and leveraging the information available through
the organization’s existing field coaching process. Managers received a summary report for their specific territories—which
typically included 8-10 sales representatives. This report allowed individual managers to follow a similar development
planning process – one that is focused on their local sales team.
O r g a n i z a t i o n a l R e s u l t s
Based on the aggregated data set, an overall, systemic analysis was conducted. Using structural equations modeling
(a sophisticated statistical technique with many advantages over multiple regression and correlation analyses), sales
representatives’ competencies were examined as drivers of territory market share and script writing volume, the primary
business outcomes for sales representatives. Using this technique, manager and physician responses were statistically
linked to the previous six-month territory market share and script volume for each sales representative. This analysis
allowed the organization to prioritize individual behaviors that have a cause-effect impact on the desired outcome (e.g.,
market share). The results of the linkage analysis are depicted below in Figure 2.
P r o d u c t & S t a t e
D i s e a s e E x p e r t i s e
M a n a g e d C a r e
E x p e r t i s e
C o n s u l t a t i v e
S e l l i n g S k i l l s
2
3
4
M a r k e t S h a r e
X Y Z S c r i p t
W r i t i n g
. 5 4 *
. 2 9 *
. 1 0 *
. 1 9 *
. 1 6 *
. 6 0 *
F i g u r e 2 .
Findings and Implications:
• Evidence-Based Selling Skills are a significant, causal driver of Script Writing and Market Share.
• Evidence-Based Selling Skills, in turn, are driven by Product & Disease State Expertise, Managed Care Expertise,
and Consultative Selling Skills.
• Prioritizing initiatives that target the key drivers listed above will lead to improvements in sales performance among
XYZ Sales Representatives.

The results of the linkage analysis identified Evidence-Based Selling® skills as the only statistically significant driver
of prescription market share and script volume. The results identified Product & Disease-State Expertise, Managed
Care Expertise and Consultative Selling Skills key drivers of Evidence-Based Selling® competence in the sales force.
Furthermore, the individual behaviors comprising the Evidence-Based Selling® competency were examined in terms
of their impact on prescription market share and script volume. The items in Table 1 have the largest statistical impact
on prescription market share and script volume. The results of this analysis provide insights as to the areas in which
investments in training and development are likely to yield the greatest return.
R E T U R N O N I N V E S T M E N T
T r a i n i n g I n t e r v e n t i o n s
The systemic analysis allowed the sales organization to assess current sales representatives’ capabilities and prioritize
training & development interventions. To address key needs in the areas of Evidence-Based Selling®,T Product &
Disease-State Expertise, Managed Care Expertise and Consultative Selling Skills the organization implemented several
training interventions at the organization and individual level.
I m p a c t o f T r a i n i n g I n t e r v e n t i o n s
The 360 process was executed again 18 months later. The table below depicts the competency changes found
following the training interventions. Based on the improvements in competency ratings, it was estimated that the training
interventions resulted in a $1.4 million impact on script volume. The average script volume went from 299 to 371 in the 2
year period. In addition to the impact on the organization’s training strategy, the results of the study were used to adjust
the brand strategy of the product. Overall, the results of the 360 were found to have numerous applications and impacts
throughout the organization.
T a b l e 1 .
Priorities determined based on the magnitude of the relationship with script writing and market share.
P r i o r i t y * C o m p e t e n c y
Evidence-Based Selling Skills
Product & Disease State Expertise
Managed Care Expertise
Consultative Selling Skills
I t e m
Easily discusses concerns about bias in XYZ's clinical
studies. Accurately answers questions about clinical
study methodology
Possesses broad knowledge of the therapeutic
market. Articulates all potential effects that XYZ can
have on patients depending on therapy paradigm and
patient profile
Understands the specifics of managed care plans.
Effectively finds solutions to managed care challenges
and restrictions
Understands physicians' unique approaches to treating
the disease state and provides solutions based on
needs. Asks pertinent and insightful questions during
interactions to identify customer needs.
1
2
3
4

T a b l e 2 .
Rating Scale: 1 = Strongly Disagree; 2 = Disagree; 3 = Somewhat Disagree; 4 = Neither Agree nor Disagree; 5 = Somewhat
Agree; 6 = Agree; 7 = Strongly Agree
B E S T P R A C T I C E S I N M U L T I - R A T E R A S S E S S M E N T D E S I G N A N D E X E C U T I O N
There is no shortage of applied research examining how to effectively design and implement multi-rater assessments in
organizations. Based on our review of this research, we offer several execution-focused recommendations to help HR
practitioners implement or improve their existing multi-rater assessment processes. Additional recommendations and
details may be found in our book, Business-Focused HR: 11 Processes to Drive Results.
A s s e s s O r g a n i z a t i o n a l R e a d i n e s s a n d I d e n t i f y C h a m p i o n s
Multi-rater assessments will be most effective if implemented at an appropriate time. Therefore, before implementing
a multi-rater assessment, it is important to consider the current organizational environment/culture as well as the
organization’s performance management history. If the organization is in the middle of a stressful time – such as
during a downsizing or extensive budget cuts – then a multi-rater assessment may be perceived by employees as a
tool for determining future workforce/budget cuts and they will not fully trust or buy into the process. As a result, the
developmental intentions of the assessment will likely not be realized, and employee morale may further be harmed.
Similarly, prior efforts to implement performance management processes should be evaluated for insights into what worked
well and what could have been improved.
I n t e g r a t e w i t h E x i s t i n g P r o g r a m s
Multi-rater assessments are just one aspect of any organization’s performance management process, which may also
include such components as individual goal-setting, training/development, and team-building. To reap the greatest benefit
from multi-rater assessments, it is thus important that they be well-integrated with existing practices. For instance, multi-
rater assessment data may be used to identify gaps in managerial competencies and this information may be used to
enhance training and development practices. Similarly, multi-rater feedback may be incorporated with career development
conversations and broader succession planning initiatives. Such alignment will help to ensure swift adoption of the process
and establish it as an integral component of the organization’s overall performance management system.
C o m p e t e n c y
Account Management
Advanced-Level Selling Skills*
Consultative Selling Skills*
Product Disease State Knowledge*
Evidence-Based Selling Skills*
Managed Care Expertise
Relationship Building
Sales Representative Overall
2 0 1 1
A v e r a g e
6.65
6.55
6.59
6.65
6.56
6.64
6.70
6.61
2 0 1 2
A v e r a g e
6.64
6.59
6.47
6.51
6.34
6.42
6.64
6.53
2 0 1 2
A v e r a g e
+.01
- .04
+ .12
+ .14
+ .22
+ .22
+ .06
+ .08
S i g n i f i c a n t
C h a n g e
NO
NO
YES
YES
YES
YES
NO
NO

E n s u r e R e l i a b i l i t y a n d V a l i d i t y o f t h e I n s t r u m e n t
Multi-rater assessments are generally comprised of sets of questions or items intended to measure particular competencies
(e.g., Relationship Building, Integrity). When developing a multi-rater assessment, it is essential to analyze the extent to
which the items being used are a valid and reliable measure of each competency and whether the sets of items predict
performance outcomes. You do not need to be an expert on statistics; however, it is important to be an educated consumer
of multi-rater assessment products and ask for relevant information in the areas of:
• Content Validity
• Discriminant Validity
• Criterion-Related Validity
• Internal Reliability
Assessing the reliability and validity of your multi-rater assessment instrument will help to ensure that you have a legally
defensible instrument that measures important aspects of individual performance and predicts critical business outcomes.
C l a r i f y t h e R o l e s a n d R e s p o n s i b i l i t i e s o f T h o s e I n v o l v e d
Many individuals are involved in the design and implementation of an organization-wide multi-rater assessment process.
In addition to HR professionals and organizational leaders/managers, organizations will often partner with an external
consulting firm to design and effective process and build the assessment framework and/or tool. With so many stakeholders
and the inherent interdependence of the tasks to be executed it is essential that everyone involved understand their role in
the process. In particular, it is important to clarify the following:
• Who will be responsible for administering the process and providing performance feedback (e.g., one-on-one feedback,
facilitated group sessions)
• The role that employees themselves play in the process
• Who will deal with technical aspects of the process
• Who will be responsible for evaluating the overall multi-rater assessment process
F o c u s o n F e e d b a c k a n d D e v e l o p m e n t P l a n n i n g
HR professionals play an important role in helping employees and their managers to interpret multi-rater feedback reports
and build development plans. Training sessions may be used to help prevent employees from becoming overwhelmed by
their feedback or misinterpreting their results and misdirecting their development efforts. Depending on the size of the
organization, these sessions may take the form of one-on-one meetings, small group discussions, or training webinars.
Regardless of the session format, participants should learn the following: 1.) the business case for the multi-rater
assessment process, 2.) the overall organization results, 3.) how to interpret their feedback report, and 4.) how to build
an action plan based on their results. In addition, HR professionals should make themselves available during or after
the training session to respond to questions and concerns. Keep in mind that receiving feedback – particularly negative
feedback – can be difficult for some individuals. HR professionals and managers across the organization should thus be
prepared to deal with the reactions of feedback and help them to turn their surprise and/ or anger into acceptance.
U s e t h e M u l t i - R a t e r A s s e s s m e n t f o r t h e R i g h t P u r p o s e s
One of the more controversial issues surrounding multi-rater assessments is the question of whether to use them exclusively
for development or to also use them as the basis for personnel decisions. Research indicates that rating quality and

accuracy increase when multi-rater assessments are used purely for development purposes. When employees know that
important personnel decisions (e.g., promotions, bonuses) will be made on the basis of their assessment results, they are
likely to select more lenient raters. Additionally, raters themselves may inflate or deflate their scores depending on their
relationship with the individual being assessed. The most useful ratings will thus be obtained if the assessment results are
used only to guide individual training and development.
E v a l u a t e t h e E f f e c t i v e n e s s o f Y o u r M u l t i - R a t e r A s s e s s m e n t P r o c e s s
The effectiveness of the multi-rater assessment process should be evaluated on a regular basis. The goal of evaluation is to
determine if you are meeting the objectives established earlier and identify any aspects of the assessment process – from
assessment tools and technologies to follow-up coaching and action planning – that may be in need of revision. Evaluation
often takes the form of surveys of those who participated in the assessment process coupled with interviews and/or
observations. The most useful evaluations go beyond simply gauging satisfaction with the multi-rater assessment process to
linking the process to meaningful business outcomes (e.g., individual performance improvement, ROI).
T H E B E N E F I T S O F M U L T I - R A T E R F E E D B A C K
Multi-rater assessments have become an increasingly common HR practice due to several advantages that they offer
over traditional performance management strategies (i.e., top-down ratings). Specifically, multi-rater assessment and
feedback processes offer: 1.) greater accuracy, 2.) enhanced face validity, 3.) richer insights into individual performance,
4.) improvements in communication, and 5.) practicality and cost-effectiveness. Each of these advantages is discussed in
greater detail below.
1 . A c c u r a c y
Traditional performance management systems rely on individual managers to assess the capabilities of their employees.
Although managers’ insights into their employees’ performance are a critical component of the performance feedback
process, relying solely on managers limits the breadth of perspectives obtained and reduces the accuracy of the
performance ratings. In contrast, multi-rater assessments take into account the perspectives of several key stakeholders
(e.g., self, manager, peers, subordinates, etc.), thus providing a more complete picture of the employee’s skills, behaviors,
and performance. Additionally, ratings are generally obtained anonymously, thereby allowing raters to provide more honest
and accurate assessments of the focal individual’s performance.
2 . F a c e V a l i d i t y
Multi-rater feedback has proven to be more accepted by participants than manager evaluations alone largely because
including multiple raters and basing performance evaluations on a comprehensive competency model enhance the face
validity of the process. Employees realize that their manager is not around to personally witness them performing every
aspect of their job. Including peers, subordinates, customers, etc. in the rating process helps participants to feel that their
performance has been fairly evaluated across all aspects of their job, thus enhancing their acceptance of the feedback and
reducing the anxiety that may arise when relying solely on the manager for performance ratings. Additionally, when the
ratings from multiple sources are based on a comprehensive competency model that truly represents the employees’ job
responsibilities, the feedback becomes very relevant and actionable.
3 . R i c h e r I n s i g h t s
Feedback from multiple sources provides participants rich insights into how their skills, behaviors, and performance
are perceived by various groups of stakeholders. Such feedback provides valuable information about variations in the
participant’s behavior as well as ways in which the same behaviors may be perceived differently by each stakeholder
group. Knowledge of these variations allows participants to make more meaningful changes in their behavior and improve
perceptions across stakeholders.

4 . I m p r o v e d C o m m u n i c a t i o n
Another benefit of multi-rater feedback processes is that they create opportunities for cross-level communications about
performance. As performance feedback from multiple sources becomes ingrained in the organizational culture, participants
begin to feel more comfortable speaking openly about their performance with not only their managers but also with other key
stakeholders (e.g., subordinates, peers).
5 . P r a c t i c a l i t y a n d C o s t - E f f e c t i v e n e s s
Multi-rater feedback processes are practical, cost-effective, and can readily be implemented by almost any organization.
They provide invaluable individual- and organizational-level diagnostic information that can readily be translated into
actionable interventions. Additionally, the cost per participant is extremely low given the significant individual and
organizational applications.
C O N C L U S I O N
This whitepaper has provided a simple roadmap to help organizations move 360 feedback processes from a simple
development tool to a strategic HR process that drives business results. Multi-rater assessment processes can provide
organizations with a new strategic resource to maximize the skills and behaviors of its employees to achieve quantifiable
business results.
6201 Fairview Road, Suite 400
Charlotte, NC, USA 28210
+1.704.227.6500
www.spisales.com | info@spisales.com

Executing a BUSINESS-FOCUSED MULTI-RATER ASSESSMENT TO DRIVE BUSINESS OUTCOMES.A Pharmaceutical company case study

Recomendados

Recomendados

Más contenido relacionado

Último

Último (12)

Destacado

Destacado (20)

Executing a BUSINESS-FOCUSED MULTI-RATER ASSESSMENT TO DRIVE BUSINESS OUTCOMES.A Pharmaceutical company case study