This study examined the impact of different scoring methods for situational judgment tests (SJTs) on validity and group differences. It found that shape-based scoring methods produced higher validities and smaller differences in scores between minority and majority groups compared to distance-based scoring methods, partially supporting the hypotheses. The results provide further evidence that shape-based scoring should be considered for SJTs used in personnel selection.

1. Prepared by Aon Hewitt | Consulting
Performance, Reward, & Talent | Assessment & Selection
The Impact of SJT Scoring Methods on
Validities and Group Differences
Kate LaPort, Christopher Huynh, & Ernest Paskey

2. Aon Hewitt | Consulting | Performance, Reward, & Talent 2
Background

3. Aon Hewitt | Consulting | Performance, Reward, & Talent 3
Background and Objective
 SJTs are valid predictors of job performance, but additional research is needed to find
SJT scoring methods which maximize scale validity and decrease adverse impact
(McDaniel & Weekley, 2012)
 McDaniel and colleagues (2011) compared distance and shape-based scoring
algorithms for SJTs that require respondents to rate each response option
– Validity: Shape-based scores control for elevation and scatter tendencies - producing
higher criterion-related validities than distance-based scoring algorithms
– Group Differences: Black respondents endorse extreme ratings on Likert scales
more frequently than White respondents - resulting in lower scores for Blacks relative
to Whites when those scores use distance-based scoring algorithms
Objective
Replicate McDaniel et al’s (2011) findings that SJT shape-based scoring methods are
associated with higher validities and lower majority-minority score differences using a
diverse sample of candidates in a high stakes testing environment.

4. Aon Hewitt | Consulting | Performance, Reward, & Talent 4
Hypotheses
Hypotheses
1
SJT shape-based scoring methods will yield higher validities than distance-based
scoring methods
2
SJT shape-based scoring methods will yield lower majority-minority mean
differences than distance-based scoring methods

5. Aon Hewitt | Consulting | Performance, Reward, & Talent 5
Method

6. Aon Hewitt | Consulting | Performance, Reward, & Talent 6
Participants and Procedure
Participants
 2,611 employees in a large, nationwide organization
 57.0% Female, 57.1% White
Procedure
 Predictive validation
 Participants responded to computer-based SJT items as part of application process
 Supervisor ratings of performance collected for employees with at least 4 months’
experience

7. Aon Hewitt | Consulting | Performance, Reward, & Talent 7
SJT Measures
Simulation
 Candidates completed a high-fidelity, computer-based simulation as part of their
employment application
 Two scenarios (Angry Customer, Multitasking) featured 4 SJT prompts assessing three
competencies:
– Stress Tolerance (2 prompts)
– Multitasking
– Customer Centricity
 Candidates were asked to rate between 7 and 9 behavioral response options on a 5-
point Likert scale (1 = Least Effective, 5 = Most Effective), and identify the most effective
response option
 Distance and shape scores were computed for each competency

8. Aon Hewitt | Consulting | Performance, Reward, & Talent 8
Performance Ratings
 Supervisors provided performance ratings of the participants’:
– Total Performance Ratings
– Overall Performance
– Adaptability
– Member Interaction
– Technology

9. Aon Hewitt | Consulting | Performance, Reward, & Talent 9
Results

10. Aon Hewitt | Consulting | Performance, Reward, & Talent 10
Hypothesis 1
Hypothesis Result
SJT shape-based scoring methods will yield higher validities than distance-
based scoring methods
Partially
Supported
Stress Tolerance Stress Tolerance 2 Customer Centricity Multitasking
Distance Shape Distance Shape Distance Shape Distance Shape
Total .09 .10 .04 .04 .06 .04 .09 .07
Overall .07 .08 .04 .04 .04 .02 .08 .06
Adaptability .09 .10 .04 .03 .05 .04 .09 .08
Member
Interaction
.09 .10 .05 .05 .05 .04 .08 .06
Technology .09 .10 .04 .03 .08 .05 .07 .06

11. Aon Hewitt | Consulting | Performance, Reward, & Talent 11
Hypothesis 2
Hypothesis Result
SJT shape-based scoring methods will yield lower majority-minority mean
differences than distance-based scoring methods
Supported
Majority Minority
Cohen’s d
Mean SD Mean SD
Stress Tolerance 1
Distance .24 .36 .13 .44 .30
Shape .69 .26 .61 .32 .27
Stress Tolerance 2
Distance -.03 .34 -.07 .39 .11
Shape .48 .31 .45 .32 .10
Customer
Centricity
Distance .10 .29 .04 .32 .20
Shape .57 .23 .53 .26 .16
Multitasking
Distance .25 .43 .20 .45 .11
Shape .66 .31 .64 .33 .06

12. Aon Hewitt | Consulting | Performance, Reward, & Talent 12
Summary

13. Aon Hewitt | Consulting | Performance, Reward, & Talent 13
Summary
 Designed to replicate McDaniel et al. (2011) findings in a diverse sample of candidates in
a high stakes testing environment
 Results provide support for further consideration of SJT shape-based scoring methods
 The validities associated with these short and efficient (i.e., 7-9 item) SJT prompts lends
further support to the development and utilization of Likert-style response scales for SJT-
items in the use of personnel selection
Hypotheses Results
1
SJT shape-based scoring methods will yield higher validities
than distance-based scoring methods
Partially
Supported
2
SJT shape-based scoring methods will yield lower majority-
minority mean differences than distance-based scoring
methods
Supported