A brief overview of the content and psychometric information presented in the WJ IV battery technical manual.

1. Select technical and
psychometric information
Dr. Kevin McGrew
Institute for Applied Psychometrics (IAP)
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

2. Detailed
information can
be found in the
extensive 369
page WJ IV
Technical
Manual

6. Technical Manual Overview
• Chapter 1 – Overview
• Chapter 2 – Test Design and Development
Procedures
• Chapter 3 – Standardization and Norm
Development Procedures
• Chapter 4 – Reliability
• Chapter 5 – Validity
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

7. One of the WJ IV guiding design objectives was for the WJ IV
to retain a focus on psychometric quality that has been
associated with the previous editions of Woodcock-Johnson
batteries:
•By providing a new, large, and nationally-representative
standardization sample of the US population;
•By updating items and simplifying test administration and
interpretation procedures;
•By augmenting the underlying scaling of speeded tests; and
•By utilizing state-of-the art data collection, test development,
and data analytic methods as models to facilitate progress in
the field of applied test development.
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

8. Contemporary CHC taxonomic model of human cognitive
abilities was used in the WJ IV test design blueprint
General
Intelligence (g)
Mathematical
achievement
(A3)
Reading
decoding (RD)
Reading
comprehension
(RC)
Reading speed
(RS)
Spelling ability
(SG)
English usage
(EU)
Writing ability
(WA)
Writing speed
(WS)
General verbal
information (K0)
Lexical
knowledge (VL)
Listening ability
(LS)
Induction (I)
General
sequential
reasoning (RG)
Quantitative
reasoning (RQ)
Memory span
(MS)
Associative
memory (MA)
Meaningful
memory (MM)
Naming facility
(NA)
Visualization
(Vz)
Visual Memory
(MV)
Spatial Scanning
(SS)
Phonetic coding
(PC)
Quantitative
Knowledge
(Gq)
Quantitative
Knowledge
(Gq)
Reading &
Writing (Grw)
Reading &
Writing (Grw)
Comp
-Knowledge
(Gc)
Comp
-Knowledge
(Gc)
Fluid
Reasoning (Gf)
Fluid
Reasoning (Gf)
Long-Term
Retrieval (Glr)
Long-Term
Retrieval (Glr)
Visual
Processing (Gv)
Visual
Processing (Gv)
Auditory
Processing
(Ga)
Auditory
Processing
(Ga)
Processing
Speed (Gs)
Processing
Speed (Gs)
Perceptual speed
(P)
Number Facility
(N)
General science
info. (K1)
Knowledge of
culture (K2)
Domain-
Specific Knw.
(Gkn)
Domain-
Specific Knw.
(Gkn)
Geography ach.
(A5)
Shading designates proposed changes in CHC model
based on analysis of WJ IV COG, OL. ACH norm data
(see Chapter 1 and Appendix A in WJ IV Technical
Manual)
Language
development
(LD)
Working memory
capacity (WM)
Attentional
Control (AC)
Speed of Lexical
access (LA)
Word Fluency
(FW)
Verbal (print)
lang. comp. (V)
Short-Term Wk
Mem. (Gwm)
Short-Term Wk
Mem. (Gwm)
Memory for
Sound Patterns
(UM)
Memory for
Sound Patterns
(UM)

9. Propositions and
Supporting Evidence for
Use of WJ IV Test Scores
Matrix Included in First
Chapter of Technical
Manual

11. Technical Manual Overview
• Chapter 1 – Overview
• Chapter 2 – Test Design and Development
Procedures
• Chapter 3 – Standardization and Norm
Development Procedures
• Chapter 4 – Reliability
• Chapter 5 – Validity
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

12. The Rasch model was used during test development,
calibration, item pool equating, and scaling

13. Expert
content,
bias, and
sensitivity
review of
items
completed

14. (etc)
Statistical item bias (DIF) analyses (sex, race, ethnicity)
informed item selection and test construction

15. Technical Manual Overview
• Chapter 1 – Overview
• Chapter 2 – Test Design and Development
Procedures
• Chapter 3 – Standardization and Norm
Development Procedures
• Chapter 4 – Reliability
• Chapter 5 – Validity
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

16. National Norm Sample
• 7,416 participants
• Preschool (664)
• K-12 (3,891)
• College/University (775)
• Adult (2,086)
• Ages 2-90+ years, Grades K.0-18.0
• 100 geographically diverse communities from 46 states and the
District of Columbia
• The Standards for Educational and Psychological Testing (aka., the
Joint Standards) (American Psychological Association, American
Educational Research Association, & National Council on
Measurement in Education, 1999; in press) guided the norming
and technical analyses
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

17. Technical Manual Overview
• Chapter 1 – Overview
• Chapter 2 – Test Design and Development
Procedures
• Chapter 3 – Standardization and Norm
Development Procedures
• Chapter 4 – Reliability
• Chapter 5 – Validity
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

18. Median
reliabilities
(across
all technical
age groups)
for WJ IV
COG clusters
General Intellectual Ability 0.97
Brief Intellectual Ability 0.94
Gf-Gc Composite 0.95
Comprehension-Knowledge 0.93
Comprehension-Knowledge- Extended 0.94
Fluid Reasoning 0.94
Fluid Reasoning- Extended 0.96
Short-Term Working Memory 0.91
Short-Term Working Memory-
Extended
0.93
Cognitive Processing Speed 0.94
Auditory Processing 0.92
Long-Term Retrieval 0.97
Visual Processing 0.86
Quantitative Reasoning 0.94
Auditory Memory Span 0.90
Number Facility 0.90
Perceptual Speed 0.93
Cognitive Efficiency 0.95
Cognitive Efficiency-Extended 0.95
Reading Aptitude A 0.93
Reading Aptitude B 0.93
Math Aptitude A 0.94
Math Aptitude B 0.95
Writing Aptitude A 0.92
Writing Aptitude B 0.93© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

19. Median reliabilities (across
all technical age groups) for WJ IV
OL clusters
Oral Language 0.90
Broad Oral Language 0.92
Oral Expression 0.89
Listening Comprehension 0.90
Phonetic Coding 0.95
Speed of Lexical Access 0.89
Vocabulary 0.93
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

20. Reading 0.95
Broad Reading 0.97
Basic Reading Skills 0.95
Reading Comprehension 0.93
Reading Comprehension-Extended 0.96
Reading Fluency 0.96
Reading Rate 0.96
Mathematics 0.96
Broad Mathematics 0.97
Math Calculation Skills 0.97
Math Problem Solving 0.95
Written Language 0.94
Broad Written Language 0.95
Basic Writing Skills 0.95
Written Expression 0.92
Academic Skills 0.97
Academic Applications 0.96
Academic Fluency 0.97
Academic Knowledge 0.95
Phoneme-Grapheme Knowledge 0.94
Brief Achievement 0.97
Broad Achievement 0.99
Median
reliabilities
(across
all technical
age groups)
for WJ IV
ACH clusters
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

21. Technical Manual Overview
• Chapter 1 – Overview
• Chapter 2 – Test Design and Development
Procedures
• Chapter 3 – Standardization and Norm
Development Procedures
• Chapter 4 – Reliability
• Chapter 5 – Validity
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

22. Sources of validity
evidence presented
•Representativeness of the WJ IV Test Content, Process,
and Construct Coverage
•Developmental Patterns of WJ IV Ability Clusters
•Internal Structure and Relations within the WJ IV
•Relationship of WJ IV Scores to Other Measures of
Cognitive Abilities, Oral Language, and Achievement
•Performance of Clinical Samples on WJ IV Measures
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

23. Sources of validity
evidence presented
•Representativeness of the WJ IV Test Content,
Process, and Construct Coverage
•Developmental Patterns of WJ IV Ability Clusters
•Internal Structure and Relations within the WJ IV
•Relationship of WJ IV Scores to Other Measures of Cognitive Abilities, Oral Language,
and Achievement
•Performance of Clinical Samples on WJ IV Measures
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

24. General
Intelligence (g)
Mathematical
achievement
(A3)
Reading
decoding (RD)
Reading
comprehension
(RC)
Reading speed
(RS)
Spelling ability
(SG)
English usage
(EU)
Writing ability
(WA)
Writing speed
(WS)
General verbal
information (K0)
Lexical
knowledge (VL)
Listening ability
(LS)
Induction (I)
General
sequential
reasoning (RG)
Quantitative
reasoning (RQ)
Memory span
(MS)
Working memory
capacity (WM)
Associative
memory (MA)
Meaningful
memory (MM)
Speed of Lexical
access (LA)
Naming facility
(NA)
Visualization
(Vz)
Visual Memory
(MV)
Spatial Scanning
(SS)
Phonetic coding
(PC)
Quantitative
Knowledge
(Gq)
Quantitative
Knowledge
(Gq)
Reading &
Writing (Grw)
Reading &
Writing (Grw)
Comp
-Knowledge
(Gc)
Comp
-Knowledge
(Gc)
Fluid Reasoning
(Gf)
Fluid Reasoning
(Gf)
Short-Term Wk
Mem. (Gwm)
Short-Term Wk
Mem. (Gwm)
Long-Term
Retrieval (Glr)
Long-Term
Retrieval (Glr)
Visual
Processing (Gv)
Visual
Processing (Gv)
Auditory
Processing (Ga)
Auditory
Processing (Ga)
Processing
Speed (Gs)
Processing
Speed (Gs)
Perceptual speed
(P)
Number Facility
(N)
General science
info. (K1)
Knowledge of
culture (K2)
Domain-
Specific Knw.
(Gkn)
Domain-
Specific Knw.
(Gkn)
Geography ach.
(A5)
WJ IV author-based contemporary CHC broad and
narrow ability WJ IV content coverage classification
Language
development
(LD)
Attentional
Control (AC)
Memory for
Sound Patterns
(UM)
Word Fluency
(FW)
Verbal (print)
lang. comp. (V)
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

25. WJ IV author classifications Independent classifications
Battery/Test name Broad Narrow Broad Narrow
Tests of Cognitive Ability
01. Oral Vocabulary Gc VL,LD Gc VL
02. Number Series Gf RQ,I Gf RQ
03. Verbal Attention Gwm WM,AC
04. Letter-Pattern Matching Gs P
05. Phonological Processing Ga/Glr PC/LA,FW
06. Story Recall Glr/Gc MM/LS Glr MM
07. Visualization Gv Vz Gv Vz
08. General Information Gc K0 Gc K0
09. Concept Formation Gf I Gf I
10. Numbers Reversed Gwm WM Gwm WM
11. Number-Pattern Matching Gs P Gs P
12. Nonword Repetition Ga/Gwm PC,UM/MS
13. Visual-Auditory Learning Glr MA Glr MA
14. Picture Recognition Gv MV Gv MV
15. Analysis-Synthesis Gf RG Gf RG
16. Object-Number Sequencing Gwm WM Gwm WM
17. Pair Cancellation Gs/Gwm/Gv P/AC/SS Gs P
18. Memory for Words Gwm MS Gwm MS
Expert Consensus Evidence: Comparison of WJ IV author and independent
broad and narrow CHC classifications of WJ III tests retained in WJ IV
Independent classifications based on Flanagan et al. (2013) and Flanagan et al. (2006).
(Reported for all COG, OL and ACH tests)

26. Test construct, process and content analysis
presented for all COG, OL and ACH tests in
the technical manual
(Reported for all COG, OL and ACH tests)

27. Figural-visual
Speed-fluency
Quantitative-numeric
Reading-writing
Auditory-linguistic
-2 -1 0 1 2
-2
-1
0
1
2
ORLVOC
NUMSER
VRBATN LETPAT
PHNPRO
VAL VISUAL
GENINF
CONFRM NUMREV
NUMPAT
NWDREP
STYREC
PICREC
ANLSYN
OBJNUM
PAIRCN
MEMWRD
PICVOC
O
R
LCM
P
SEGMNT
RPCNAM
SENREP
UNDDIR
SNDBLN
RETFLU
SNDAWR
LWIDNT
APPRO
B
SPELL
PSGCMP
CALC
WRTSMP
WRDATK
ORLRDG
SNRDFL
MTHFLU
SNWFL
RDGREC
NUMMAT
EDIT
WRDFLU
SPLSND
RDGVOC
SCI
SOCHUM
MEMNAM
VRBANL
VISCLO
NUMSEN
Gs-Cog
Gs-Ach
Glr-LA
Ga
Grw
Gwm-
MS
Gv
Glr-MM
Gwm-
WM
Gc
Gq/Gf-RQ
Gf
Use of MDS
Guttman
Radex analysis to
evaluate content
validity
Figure is for WJ IV
norm study
participants from
ages 9-13 (model
development
sample; n = 785).
Results for 6 age
groups presented in
technical manual

28. Sources of validity
evidence presented
•Representativeness of the WJ IV Test Content, Process, and Construct Coverage
•Developmental Patterns of WJ IV Ability Clusters
•Internal Structure and Relations within the WJ IV
•Relationship of WJ IV Scores to Other Measures of Cognitive Abilities, Oral Language,
and Achievement
•Performance of Clinical Samples on WJ IV Measures

30. Sources of validity
evidence presented
•Representativeness of the WJ IV Test Content, Process, and Construct Coverage
•Developmental Patterns of WJ IV Ability Clusters
•Internal Structure and Relations within the WJ IV
•Relationship of WJ IV Scores to Other Measures of Cognitive Abilities, Oral Language,
and Achievement
•Performance of Clinical Samples on WJ IV Measures
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

31. Stage 3:
Confirmatory
structural
model cross-
validation
phase
Stage 2:
Exploratory structural model
generation (MG) and evaluation
Ages 3-5 A
(n = 209)
Ages 3-5 B
(n = 208)
Ages 6-8 A
(n =412)
Ages 6-8 B
(n = 411)
Ages 9-13 A
(n = 785)
Ages 9-13 B
(n = 787)
Ages 14-19 A
(n = 842)
Ages 14-19 B
(n = 843)
Ages 20-39 A
(n = 625)
Ages 20-39 B
(n = 626)
Ages 40-90+ A
(n = 571)
Ages 40-90+ B
(n = 575 )
Model
Development
(MD)
samples (A)
Model
Cross-Validation
(MCV)
samples (B)
Stage 1:
Split-sample
random sample
generation
Three-stage internal/ structural validity procedures for WJ IV
battery
+
+
Cluster analysis
(CA)
Multidimensional
scaling analysis
(MDS)
Principal
component
analysis
(PCA)
Stage 2A
Exploratory structural
analysis in Sample A
Review of
prior WJ,
WJ-R, WJ III
structural
validity
research
Specify initial
confirmatory
factor analysis
model (CFA)
Review of
contemporary
CHC and
cognitive
neuroscience
research
CFA of MG model
result from MD (A)
sample in MCV (B)
sample(s)
Yes
Most
plausible
and best
fitting
odel?
Model adjustment(s)
No
Stage 2B
Specify initial CFA
models
Stage 2C
Evaluate and adjust
CFA
model

32. Stage 3:
Confirmatory
structural
model cross-
validation
phase
Stage 2:
Exploratory structural model
generation (MG) and evaluation
Ages 3-5 A
(n = 209)
Ages 3-5 B
(n = 208)
Ages 6-8 A
(n =412)
Ages 6-8 B
(n = 411)
Ages 9-13 A
(n = 785)
Ages 9-13 B
(n = 787)
Ages 14-19 A
(n = 842)
Ages 14-19 B
(n = 843)
Ages 20-39 A
(n = 625)
Ages 20-39 B
(n = 626)
Ages 40-90+ A
(n = 571)
Ages 40-90+ B
(n = 575 )
Model
Development
(MD)
samples (A)
Model
Cross-Validation
(MCV)
samples (B)
Stage 1:
Split-sample
random sample
generation
Three-stage internal/ structural validity procedures for WJ IV
battery
+
+
Cluster analysis
(CA)
Multidimensional
scaling analysis
(MDS)
Principal
component
analysis
(PCA)
Stage 2A
Exploratory structural
analysis in Sample A
Review of
prior WJ,
WJ-R, WJ III
structural
validity
research
Specify initial
confirmatory
factor analysis
model (CFA)
Review of
contemporary
CHC and
cognitive
neuroscience
research
CFA of MG model
result from MD (A)
sample in MCV (B)
sample(s)
Yes
Most
plausible
and best
fitting
odel?
Model adjustment(s)
No
Stage 2B
Specify initial CFA
models
Stage 2C
Evaluate and adjust
CFA
model
All Stage 2A
(exploratory)
results reported
in detail in the
technical manual

33. Wards cluster analysis of ages 9-13 year sample (model development sample A; n=785)
NUMREV
OBJNUM
VRBATN
MEMWRD
SENREP
NWDREP
UNDDIR
MEMNAM
WRDATK
SPLSND
SNDAWR
SEGMNT
PHNPRO
SNDBLN
VAL
VISUAL
ANLSYN
CONFRM
VISCLO
PICREC
RPCNAM
RETFLU
ORLCMP
VRBANL
HUM
PICVOC
ORLVOC
GENINF
SOC
SCI
STYREC
WRTSMP
RDGREC
NUMSEN
APPROB
CALC
NUMSER
NUMMAT
SPELL
LWIDNT
PSGCMP
RDGVOC
EDIT
ORLRDG
MTHFLU
WRDFLU
SNRDFL
SNWRFL
NUMPAT
LETPAT
PAIRCN
Gc
Glr-MM/Grw
Gq/Gf-RQ
Grw
Gs
Gs-ACH
Gs-COG
Acquired knowledge systems
Gv
Gf
Gf+Gv
Glr-LA
Gwm-
WM
Gwm-
MS Gwm
Ga-PC
Ga-PC/Grw
Ga
Ga+Gwm (sequential/serial processing)
Cognitive Operations:
Controlled/deliberate/system 2
General intelligence (g)
Bold font = Broad or general
CHC abilities
Underlined font= Possible
intermediate level stratum
dimensions
Groupings to the left of the
vertical dashed lines
represent the groupings
interpreted as representing
valid CHC abilities. Higher-
order groupings to the left
of the dashed line are
potentially interesting
hypothesized intermediate
ability dimensions.
Glr-NA
Glr (Retrieval Fluency)
Cognitive Operations:
Automatic/system 1

34. At most ages
7-10 principal
component
exploratory
solutions
were
extracted and
reviewed

35. At most ages
7-10 principal
component
exploratory
solutions
were
extracted and
reviewed

36. At most ages
7-10 principal
component
exploratory
solutions
were
extracted and
reviewed

37. -2 -1 0 1 2
-2
-1
0
1
2
ORLVOC
NUMSER
VRBATN LETPAT
PHNPRO
VAL VISUAL
GENINF
NUMREV
NUMPAT
NWDREP
STYREC
PICREC
ANLSYN
OBJNUM
PAIRCN
MEMWRD
PICVOC
O
R
LCM
P
SEGMNT
RPCNAM
SENREP
UNDDIR
SNDBLN
RETFLU
SNDAWR
LWIDNT
APPRO
B
SPELL
PSGCMP
CALC
WRTSMP
ORLRDG
SNRDFL
MTHFLU
SNWRFL
RDGREC
NUMMAT
EDIT
WRDFLU
SPLSND
RDGVOC
SCI
SOCHUM
MEMNAM
VRBANL
VISCLO
NUMSEN
Gs-Cog
Gs-Ach
Glr-LA
Ga
Grw
Gv
Glr-MM
Gc
Gq/Gf-RQ
Gwm-
WM
Gwm-
MS
WRDATK
MDS
(Guttman
Radex) of
ages 9-13
year sample
(model
development
sample A;
n=785 )
Gf
CONFRM

38. A
Two final WJ IV cross-
validated models reported
across age groups
A = Broad CHC top-down
model.
Results presented for all
cross-validation samples

39. Two final WJ IV cross-
validated models finalized
across age groups
B = Broad+narrow CHC
bottom-up model.
Results presented for all
cross-validation samples
B

40. Stage 3:
Confirmatory
structural
model cross-
validation
phase
Stage 2:
Exploratory structural model
generation (MG) and evaluation
Ages 3-5 A
(n = 209)
Ages 3-5 B
(n = 208)
Ages 6-8 A
(n =412)
Ages 6-8 B
(n = 411)
Ages 9-13 A
(n = 785)
Ages 9-13 B
(n = 787)
Ages 14-19 A
(n = 842)
Ages 14-19 B
(n = 843)
Ages 20-39 A
(n = 625)
Ages 20-39 B
(n = 626)
Ages 40-90+ A
(n = 571)
Ages 40-90+ B
(n = 575 )
Model
Development
(MD)
samples (A)
Model
Cross-Validation
(MCV)
samples (B)
Stage 1:
Split-sample
random sample
generation
Three-stage internal/ structural validity procedures for WJ IV
battery
+
+
Cluster analysis
(CA)
Multidimensional
scaling analysis
(MDS)
Principal
component
analysis
(PCA)
Stage 2A
Exploratory structural
analysis in Sample A
Review of
prior WJ,
WJ-R, WJ III
structural
validity
research
Specify initial
confirmatory
factor analysis
model (CFA)
Review of
contemporary
CHC and
cognitive
neuroscience
research
CFA of MG model
result from MD (A)
sample in MCV (B)
sample(s)
Yes
Most
plausible
and best
fitting
odel?
Model adjustment(s)
No
Stage 2B
Specify initial CFA
models
Stage 2C
Evaluate and adjust
CFA
model
All final Stage 2C
(exploratory) and
Stage 3
(confirmatory cross-
validation) results
reported in detail in
the technical manual

41. Test EFA
Other exp.
methods CFA Cross-validation
WJ III No No Yes No
SB5 No No Yes No
DAS-II No No Yes No
KABC-II No No Yes (in “exploratory
manner”)
No
WAIS-IV No No Yes No
WISC-IV Yes No Yes Yes: Factor score (7)
congruence in random CV
sample of 440
WJ IV Yes Yes: MDS
and cluster
analysis
Yes: Exploratory
model generating
CFA (MD samples)
and CV CFA (CV
samples)
Yes: Randomly split
samples at six age groups.
CV of final MD models in
CV sample
Structural validity method comparisons across 7
intelligence batteries
© Institute for Applied
Psychometrics; Kevin McGrew 6-
19-14

42. Two final WJ IV cross-validated models across age groups: A = Broad CHC top-down model; B = Broad+narrow CHC bottom-up model
A B
99.5 % of test and
latent factor
loadings cross-
validated (were
significant in all
age-differentiated
samples cross-
validation B
samples)
98.6 % of test and
latent factor
loadings cross-
validated (were
significant in all
age-differentiated
samples cross-
validation B
samples)

43. Sources of validity
evidence presented
•Representativeness of the WJ IV Test Content, Process, and Construct Coverage
•Developmental Patterns of WJ IV Ability Clusters
•Internal Structure and Relations within the WJ IV
•Relationship of WJ IV Scores to Other Measures
of Cognitive Abilities, Oral Language, and
Achievement
•Performance of Clinical Samples on WJ IV Measures
© Institute for Applied Psychometrics; Kevin McGrew 6-19-14

44. 15 independent WJ IV concurrent validity studies
differentiated by age/grade
WJIV COG compared to:
• WISC-IV, WAIS-IV, WPPSI-III, KABC-II, DAS-II
WJIV OL compared to:
• CELF-F, PPVT-4, CASL, OWLS
WJIV ACH compared to:
• KTEA-II, WAIT-III, OWLS-WE

45. Sources of validity
evidence presented
•Representativeness of the WJ IV Test Content, Process, and Construct Coverage
•Developmental Patterns of WJ IV Ability Clusters
•Internal Structure and Relations within the WJ IV
•Relationship of WJ IV Scores to Other Measures of Cognitive Abilities, Oral Language,
and Achievement
•Performance of Clinical Samples on WJ IV
Measures