2. Test Development Process
1. Statement of Goals
2. Content Outline
3. Table of Specifications
4. Item Selection
5. Item Construction
6. Composition of Instructions
7. Development of Answer Sheets
8. Construction of Answer Keys
9. Test Administration
10. Test Revision
4. Two Important Characteristics of Tests
Reliability
– consistency
• free from extraneous sources of error
Validity
– how well a test measures what it is sup
posed to measure
5. Formative vs. Summative Tests
Formative
– monitor progress toward goals within a
course of study
Summative
– assess overall achievement of course
goals
6. A. Table of Specifications
Blue print for test
Purpose
– ensure proper emphasis given to all el
ements of a course of study
– content validity
Guide for writing items
7. Table of Specifications (2)
Start with Instructional Objectives
– Biology 30: Circulation Unit Test
1. Identifies parts of circulation system.
2. Explains function of parts in relation
to whole system.
3. Distinguishes between circulation
system and other bodily systems.
9. Need to Consider and Decide:
Length of Test
Weight to be given to each objective
Weight to be given to each level of t
axonomy
Estimate number of items in each ce
ll
12. C. Item Construction
Selected Response Test Items
– Item construction skills for valid and reli
able measures of student achievement.
Guidelines which apply to all types
Specific Suggestions for writing each type
Advantages and Disadvantages of Each
13. Guidelines for Writing Objective Items
1. Construct at appropriate level of
difficulty for examinees
2. Include Items at appropriate level of
difficulty for purpose of test.
3. Test significant elements of a
course.
4. Write independent items.
14. 5. Construct questions free from
extraneous reasons for problems.
6. Communicate the question in clear,
concise language.
7. In the correct alternatives,
paraphrase statements from the text.
8. Exclude clues to correct answer.
II. Guidelines for Writing Objective Items
15. III. Guidelines for Writing Objective Items
9. Provide one correct answer.
10. Edit the Items.
16. Alternate Response Items
Involves the selection of one of two alternati
ves
– true / false
– yes / no
– right / wrong
– fact / opinion
Mainly for Knowledge & Comprehension
– Can be written at higher levels
17. True / False
Word statements clearly. Vague or ambiguous wording
will confuse students.
Avoid overgeneralizing.
– Poor: Heavy smoking causes lung cancer. T F
– Better: Heavy smoking often causes lung cancer. T F
Avoid Trick questions.
– i.e., General Wolseley led Canadian troops to Manitoba in 1
870.
– 3 tricks: Colonel, British, Not yet a province
Do not use trivial statements to “pad out” the number of
questions and marks to arrive at a predetermined level.
18. True / False
Statements should be entirely true, or entirel
y false:
– Unacceptable:
• In King Lear, Regan ordered Gloucester’s eyes to be
plucked out and Gloucester died when he jumped off
the cliff of Dover.
– Acceptable:
• In King Lear, Regan ordered Gloucester’s eyes to be
plucked out. (T)
• In King Lear, Gloucester died when he jumped off th
e cliff at Dover. (F)
19. True / False
Avoid using universal descriptors such as “nev
er”, “none”, “always”, and “all”.
– Testwise students will recognize that there are few ab
solutes.
Avoid negative words, as they are often overloo
ked by students.
– Poor: It was not unheard of for Henry VIII to close
monasteries in England. T F
– Better: Henry VIII closed some monasteries in England.
(T)
20. True / False
Do not include two ideas in one statement unless you are
evaluating student’s understanding of cause and effect rel
ationships.
– Poor: Porpoises are able to communicate because
they are mammals. T F
– Better: Porpoises are mammals. T F
Porpoises are able to communicate. T F
Provide a “T” and “F” beside each statement and ask stud
ents to circle correct answer.
– Avoids problem of students writing illegible letters.
21. True / False & Variations
Include more false than true statments in any given test an
d vary the number of false statements from test to test.
– tendency to mark more statements true than false.
– discrimination between those who know the content an
d those who do not is greater for false expressions.
Avoid using negative statements.
– Under the demands of the testing situation, students m
ay fail to see the negative qualifier.
22. Matching Items
Consist of
– a column of premises
– a column of responses
– directions for matching the two.
Similar to multiple choice, but easier and more e
fficient to construct
Can be written to assess Knowledge, Comprehe
nsion, Application, Analysis level behaviors
23. Guidelines for Writing Matching Items
Provide clear instructions on how to indicate the corr
ect answers.
Indicate whether the same response can be used mor
e than once.
Maintain grammatical consistency within and between
columns.
– within a column: either sentence or point form
– between columns: one or the other
Ensure that any matching question appears entirely o
n one page.
24. 2. Guidelines for Writing Matching Items
Provide an unequal number of premises and res
ponses
– reduces guessing and elimination
– increases measure of comprehension
Avoid designing questions which require studen
ts to draw lines between premise and response.
– confusing for student and marker
– provide space for letter or number answers
25. 3. Guidelines for Writing Matching Items
Make sure lists are homogeneous.
– i.e., do not include items testing names, dates, and eve
nts.
– Instead, make every response plausible
Make the wording of the premises longer than th
e wording of the responses.
Identify the items in one list with numbers and th
ose in the second list with letters.
26. Example:
Directions: 1. On the line to the right of each phrase in
Column I, write the letter for the word in
Column II that best matches the phrase.
2. Each word in Column II may be used once,
more than once, or not at all.
Column I Column II
1. Name of the answer in addition
problems.
2. Name of the answer in subtracti
on problems.
3. Name of the answer in multiplic
ation problems.
4. Name of the answer in division
problems.
A. Difference
B. Dividend
C. Multiplicand
D. Product
E. Quotient
F. Subtrahend
G. Sum
27. Short Answer Test Items
Typically, the student is asked to reply with a wo
rd, phrase, name, or sentence, rather than a mor
e extended response.
– Direct Questions / Short Answer
• Who is the current Prime Minister of Canada?
– Incomplete Sentences / Fill In the Blanks
• The current Prime Minister of Canada is _____?
Items are fairly easy to construct and mark
Assess mainly knowledge, comprehension, and
some application.
28. Guidelines for Writing Short Answer Items
Questions must be carefully worded so that all s
tudents understand the specific nature of the qu
estion asked and the answer required.
– Poor: Wellington defeated Napoleon in _____ ?
– Better: In what battle fought in 1815 did Wellington
defeat Napoleon?
OR
In what year did Wellington defeat Napoleon
at Waterloo?
29. II. Guidelines for Writing Short Answer Items
Word completion or fillin questions so that missing infor
mation is at, or near the end of, the sentence. Makes rea
ding and responding easier.
– Poor: In the year ______ , Canada turned 100 years ol
d.
– Better: Canada turned 100 years old in the year _____
.
Instructions and teacher’s expectations about filling in b
lanks should be made clear. Indicate whether each blank
of equal length represents one word or several words, w
hether long blanks require sentences or phrases, and w
hether synonymous terms are accepted.
30. III. Guidelines for Writing Short Answer Items
When an answer is to be expressed in numerical units, the
unit should be stated.
– Poor: If a room measures 7 metres by 4 metres, the per
imeter is _____ .
– Better: If a room measures 7 metres by 4 metres, the pe
rimeter is _____ metres (or m).
Do not use too many blanks in completion items. The emp
hasis should be on knowledge and comprehension, not mi
nd reading!
– Consider:
In the year _____ , Prime Minister _________ signed the
__________ , which led to a ___________ which was ___
_______ .
32. 1. The capital city of Canada is
a. Vancouver
b. Montreal
c. Toronto
*d. Ottawa
Terminology: Multiple Choice
1. Stem: presents the problem
2. Keyed Response: correct or best answer
3. Distracters: appear to be reasonable answers to the exa
minee who does not know the content
4. Options: include the distracters and the keyed respons
e.
4
1
2
3
33. Ambiguity
Extrinsic
Desirable quality in
multiple choice item
s
Outside the item
Allows discriminatio
n between those wh
o know material and
those who do not.
Intrinsic
Undesirable
Should be avoided
Inside the item
poor wording
more than one answer
Even those who know con
tent have difficulty choosi
ng correct answer
34. Types of Multiple Choice Items
Correct Answer*
– Only one correct response
Best Answer
– requires examinee to select alternative cl
osest to being correct
– fine distinctions
Multiple Answer
– More than one correct or best answer
35. Interpretive Exercise
Usually begins with verbal, tabular or graphic inf
ormation which is the basis for 1 or more multipl
e choice questions.
– map, passage from a story, a poem, a cartoon
Can challenge students at various levels of unde
rstanding
– application, analysis, synthesis, evaluation
Exercise contains all information needed to ans
wer questions
Readily adaptive to the more important outcome
s of disciplines.
36. Interpretive Exercises (con’t)
Examples
If student answers incorrectly it is because the
y have not mastered the thinking or reasoning r
equired by the question, NOT because they fail
ed to memorize background information.
Math questions: give students the formulas, te
st ability to apply concepts, rather than ability t
o memorize formulas.
37. Guidelines for Writing:
Multiple Choice Items
State stem in the form of a question.
– Weak
– Canada is
• a) a country
• b) where you live
• c) between the Pacific and Atlantic oceans
• d) all of the above
– Better
– Between which two oceans is Canada located?
• a) Atlantic and Indian
• b) Atlantic and Pacific
• c) Pacific and Indian
38. Guidelines for Writing: Stems
Place most of the subject matter in the Stem
– ensures full statement of problem
Eliminate extraneous material from the Stem
– goal is to measure student achievement, not to present ne
w material
– maximize use of time for demonstrating understanding, no
t reading ability
Avoid Negatively phrased Stems
– students may miss the qualifier
– use only when learning outcome requires this type of diffe
rentiation
39. I. Guidelines for Writing: MC
Ensure similarity among alternatives wit
h regard to:
– grammatical structure
– length
– mode of expression
Grammatical errors provide unintentiona
l clues to the answer
When in doubt, students will select the lo
ngest alternative as the correct answer
40. Example: Length of Alternatives
Neurotics are more likely than psychotics to
a. be dangerous to society
b. have delusional symptoms
c. be dangerous to themselves
*d. have insight into their own inappropriate
behavior but nevertheless feel rather
helpless in terms of dealing with their
difficulties
41. II. Guidelines for Writing: MC
Make one of the alternatives the most clearly
correct or best answer
– exception: multiple answer form
– reduces intrinsic ambiguity
– reduces frustration during test
Make distracters plausible
– desire to attract students who really do NOT know the an
swer to the question
– create distracters from elements of the correct response
– improves reliability of item
42. Example: Undemanding Distracters
Mickey Mouse’s two nephews are
a) Huey, Dewey and Louie
b) Clarabelle Cow
*c) Morty and Ferdy
d) Abbott and Costello
The Role of Humour?
43. III. Guidelines for Writing: MC
Avoid parallel language between the Stem and
the Correct Response
– gives clues to keyed response
– emphasizes testwiseness, not knowledge
Randomly distribute answers across the altern
ative positions
– inexperienced test writers emphasize “b” and “c” alternati
ves (hide the answer!!)
– do NOT use an interpretable order of keyed responses
44. IV. Guidelines for Writing: MC
Use qualifiers such as “all of the ab
ove” and “none of the above” spari
ngly
– testwise students will use process of eli
mination to select answer
– do NOT use to “pad out” the distracters
because you cannot think of another on
e.
45. Advantages of Multiple Choice Items
allow more adequate sampling of content.
tend to more effectively structure the problem t
o be addressed
items can be more efficiently and reliably score
d than supply items
different response alternatives can provide diag
nostic feedback (item analysis)
items can be constructed to address various le
vels of cognitive complexity
46. Disadvantages of Mult.Choice Items
difficult & time consuming to construct good ite
ms
– leads to emphasis on other selected response item types
can lead the instructor to favour simple recall o
f facts
high degree of dependence on student’s readi
ng and instructor’s writing ability
– can be difficult to achieve clarity of expression
measuring synthesis and evaluation can be dif
ficult
inappropriate for measuring outcomes that req
uire skilled performance
