Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Psychometrics Slides
1. Psychometrics 101
Emma Rush
Session for the Legal Education and Training
Group
Emma Rush Consulting February 2011
2. Agenda
Gain a better understanding of the ‘added value’ a
psychometric can bring within a legal environment
Get a clear overview of the six main instruments on
the market and when they can be used
Understand the resource issues of psychometrics:
the cost, the time required, the trained personnel
needed
Develop your own case, if required, for introducing
psychometrics to all or any of the ‘people processes’
Emma Rush Consulting
3. Starting with me
As a consultant:
Hogan™, MBTI Step II® and OPQ32r™ for coaching and
career consulting
Previous law firm:
16PF® for Assessment for Development Centre
MBTI Steps I & II ® for team workshops
FIRO ® for Leadership programme
FSI:
16PF ® mapped on to a competency framework for partner
recruitment and partnership track
FIRO® for Assertiveness
Destiny for secretarial recruitment
Emma Rush Consulting
4. Brief history of psychometrics
Began in late C19th with “intelligence” testing
Francis Galton, UK: Anthropometrics – taking a
mathematical approach to the measure of individual
differences – eugenics
Charles Spearman, USA 1910 – contributed ‘g’ or
the measurement of general intelligence factor
And “factor analysis” – the means by which
psychometric tests are found to be valid
Emma Rush Consulting
5. Brief history of psychometrics
1917 Stanford-Binet test: first standardised
test of intelligence
1949: First publication of the 16PF by
Raymond Cattell (Spearman’s student)
1962: MBTI first published – type-based
instrument; four preference scales
Emma Rush Consulting
6. Brief history of psychometrics
1963 First appearance of the “Big Five” of
personality factors
1970: Cronbach (Cronbach’s Alpha):
emphasises the importance of validity – being
able to predict accurately what a person will
do
Emma Rush Consulting
7. Brief history of psychometrics
1984: OPQ32 introduced, published by SHL
R&D by Peter Saville and Roger Holdsworth
1986: Hogan’s Personality Inventory first
published
2004 Saville and Holdsworth ousted from
SHL’s board
2008 Saville’s Personality Questionnaire /
Wave introduced
Emma Rush Consulting
8. Key concepts
Reliability: The degree to which an
instrument measures the same way each time it
is used under the same conditions with the
same subjects – its “consistency”
Validity: The extent to which you actually
are measuring what you are professing to
measure
Type vs. Trait: Type gives you a preference,
trait measures how you compare to a wider
population
Emma Rush Consulting
9. Where to use psychometrics
Used in thousands of organisations at all
stages of the employee “life cycle”
We’re looking at three: Selection;
Development ; Promotion
Emma Rush Consulting
13. Why select (or train) with
psychometrics?
To make the business case you first need
to know:
What it currently costs (directly and indirectly) to
recruit, train and retain someone so you can
measure/prove an improvement
Exactly what you are looking for – job criteria,
competency framework
What outcomes you require from training
interventions
Emma Rush Consulting
14. In addition testing…
Allows for better and clearer discrimination between
candidates – “beyond the good egg factor”
Supports your diversity initiatives: more transparent,
fairer and no adverse impact
Allows a better fit with your job criteria or
competency framework
Administrative convenience (depending on your
resources)
Costs and development time are reasonable
Emma Rush Consulting
15. What do you test?
Five key areas to measure How do they get measured?
Knowledge Areas on which traditional or
unstructured approaches
Experience concentrate – psychometrics can
still assess and assist
Behaviours Areas where online tests and
structured approaches can assess
Personality these areas and increase validity
Capability of recruitment
In descending order of ease to
Motivation change – motivation hardest
Emma Rush Consulting
16. Intellectual Ability
Job-based or reasoning tests – used
frequently at trainee level, but not at higher
and more expensive level
Standard intellectual ability tests:
Verbal reasoning
Numerical reasoning
Thinking style
Emma Rush Consulting
18. A word of warning to begin!
“..it seems that users of some
questionnaires become attached to the
tests that “look right” or appear to be
appropriate, which is known as faith
validity..” (Saville, 1975)
For every argument there is a counter-
argument
Emma Rush Consulting
19. The Big , in in test terms
OPQ32r®
The 16PF5 ®
NEO PI-R™
Hogan Personality Inventory and
Development Survey™
Saville Wave Professional Styles®
Emma Rush Consulting
20. How valid are they?
Figures taken from Saville’s report How valid is your
questionnaire?
High
validity
Moderate
validity
Chance
Emma Rush Consulting
21. A time comparison
Instrument No of Questions Typical Completion time
OPQ32i 416 60 minutes
NEO-PI-R 240 40 minutes
Wave Professional 216 40 minutes
Styles
Hogan 206 30 minutes
16PF5 185 30 minutes
Saville’s Personality 72 13 minutes
Questionnaire
Emma Rush Consulting
22. Tests to avoid
Ipsative Tests
Tests without appropriate norms
Tests with low reliability estimates
Old tests that have not been re-evaluated in
the last 10 years.
Glossy, packaged tests with no psychometric
details
Emma Rush Consulting
23. And not to be used for selection
DISC: “The test suffers from questionable
reliability and unknown validity”
MBTI: a “type” based instrument so
without norm group validity
Emma Rush Consulting
24. Costs
Using psychometrics may have a long
term benefit but also has a short term cost
Per instrument – from £12 to £298 per report
Time: an hour’s minimum feedback
Online – no additional cost to set up
Reputational risk
Emma Rush Consulting
25. Which tests to use?
OPQ32r™
Pros: an “Occupational” questionnaire so about the
“you at work”
“r” is the latest version, and has three rather than
four item choices
Published by SHL and designed by Professor Peter
Saville
Can be mapped onto a competency framework
Multiple report options
Relatively quick to complete
Emma Rush Consulting
26. OPQ32r™
Relationships with Feelings and
People Emotion
Influence Emotion
Sociability Dynamism
Empathy
Thinking Style
Analysis
Creativity and Change
Structure
Emma Rush Consulting
27. OPQ32r™
Cons:
32 factors – do they add anything extra? They take
longer to give feedback on
Overlap between factors e.g. relationships with people /
feelings and emotion
Now quite old (1984), doesn’t cover to same degree
motivations, values, influencing style, leadership
Is it an ideal fit with professional services – norm groups
used?
Hard to get an overview of scores in the reports, more
interpretative than detailed
Emma Rush Consulting
28. 16PF®
Pros: A combination of five global factors and
sixteen primary factors – gives some granularity but
not overwhelming
Used world-wide for recruitment and development
Can be mapped onto a competency framework
Factors relate to universal traits rather than forced
into three pronged model
Reports give practitioner option as well as
interpretative reports
Relatively quick to complete
Emma Rush Consulting
29. 16PF®
Cons: Meta-perceptions: candidates frequently
report “I do this differently in social situations”
Measures drive (obliquely) but not specifically
motivations
Reasoning questions can’t be relied on as
measure of intellectual ability, particularly for
lawyers
Is it an ideal fit for professional services – norm
groups used?
Emma Rush Consulting
30. Hogan Personality Inventory/Development
Survey/Motivations, Preferences and Values Inventory
(all™)
Pros: Uniquely covers both “bright” and
“dark” side of personality
Covers what drives individuals, what derails
them and what they value in an organisation
Particularly suited for senior recruits e.g..
lateral hires, as reports based on strengths
and competencies for leaders
Emma Rush Consulting
31. Hogan Personality Inventory/Development
Survey/Motivations, Preferences and Values Inventory
(all™)
Cons: quite American – language in reports,
language in questions
Three separate questionnaires to complete
Relevance to professional services?
Relevance to more junior roles in professional
services?
Supplied in UK by smaller distributors e.g.
Mentis; Psychological Consultancy
Emma Rush Consulting
32. NEO
The NEO PI-R was developed by Costa
and McCrae
Five domains measured: Neuroticism,
Extraversion, Openness to Experience,
Agreeableness and Conscientiousness.
Emma Rush Consulting
33. NEO
Each five domains is divided into six
facets: thirty areas of personality which
predict employees’ work behaviour
including leadership style, team style,
decision-making and stress management.
240-item questionnaire either online or by
paper and pencil
The current UK Edition was published in
2004.
Emma Rush Consulting
34. Saville Consulting’s Wave ®
Professional Styles edition
To quote Peter Saville: “Saville Consulting
Wave® Professional Styles measures motives,
talents, preferred culture and competency
potential in one dynamic online questionnaire”
A wealth of detail: 4 Clusters; 12 Sections; 36
Dimensions; 108 Facets
More reliable
Quick to complete
Normed for professionals
Emma Rush Consulting
36. Ongoing development: workshops
and coaching
Specific learning e.g. leadership,
assertiveness
Coaching
Team workshops
Emma Rush Consulting
37. Why train with psychometrics?
The sheer cost of training
Have to do something 200 times to be
completely proficient in a new skill -
accelerates the process of self-awareness
Assessing motivation means you can see in
advance:
How this person likes to work
What motivates this person to work
What types of goals this person likes
Emma Rush Consulting
38. What type of intervention?
How you like to learn e.g. Honey and
Mumford’s Learning Styles Questionnaire
What motivates you – how do we reinforce
learning e.g. Hogan MPVI
Why you need to learn e.g. FIRO-B with
leadership
How you operate in a team e.g. MBTI Step II
What the real issues are in coaching e.g.
Saville Consulting Wave® Professional Styles
Emma Rush Consulting
39. Team Interventions
Three instances you may instigate a team
workshops:
Forming a team
Dealing with team performance
Team building
Emma Rush Consulting
40. FIRO-B
A unique way of looking at how a team
operates
Three “scales”, no typology, scored from 1 to
9
Inclusion; Control; Affection (or Openness)
An “Expressed” score: what you show to
others
A “Wanted” score: what you want from
others but may not show
Emma Rush Consulting
41. Team Workshops
MBTI Steps I and II
Uniquely lends itself to group feedback with a
reported type followed by self-assessed type
and ending with best fit
Step II adds factors to the four dimensions
giving more granularity and better comparison
between team members
Can be a powerful aide to raising self-
awareness
Emma Rush Consulting
42. However…
Can feel over-simple even at Step II
MBTI Step I gives only four dimensions for
comparison – does this adequately explain how
a team interacts or personality works?
Slip between “reported type” and “best fit” can
be as much as 25% e.g.. only 75% of people
confirm their MBTI results
Lacks real scientific validity
Emma Rush Consulting
43. Belbin
Dr Meredith Belbin; dates from the 1970s
Belbin’s Nine Team Roles
Plant Co-ordinator
Monitor-Evaluator Resource-
Investigator
Implementer Completer-finisher
Team worker Shaper
Specialist
Emma Rush Consulting
45. A form of selection
The same instrument should be used for
promotion purposes as for selection for
reasons of:
Consistency
Fairness
Transparency
Proper metrics of progress (from selection to
now)
Emma Rush Consulting
46. How do you make the best
decision on promotion?
The more structured methods you use the
better validity you gain
Assessment centres alone are less valid than
when combined with e.g.
Structured interviews
Psychometrics
Emma Rush Consulting
50. Training
I recommend working out which test(s) you want
to use, and then qualifying with that test
publisher
For example OPP’s Level B course qualifying
you in both MBTI® and 16PF®
Or Saville Consulting qualifying you in Wave™
Emma Rush Consulting