2. An item, also called an item set, is composed of four
questions that measure the same discrete skill or concept at
the same level of difficulty. We’ll be looking at item sets in
another module.
A question consists of a stem, a key, and three distractors.
Let’s take a closer look at these individual parts.
The Item
4. • A stem can take the form of a question or a
statement.
• Stems are best written as complete sentences. They can also
take the form of a statement with a blank at the end.
• Be careful not to overdo fill-in-the-blank questions; a given
item set should probably only contain one, if any.
Composing Stems
5. Acceptable:
A dash is a punctuation mark used to _____.
Acceptable:
Aristotle defines rhetoric as:
Better:
A dash is a punctuation mark with what function?
What function does a dash have?
How does Aristotle define rhetoric?
Aristotle’s definition of rhetoric includes what?
What does Aristotle’s definition of rhetoric include?
6. • Stems can also include a chart, table, or other graphic.
This is especially useful in courses such as business or finance, but can
fit with any subject, as with this question from a remedial reading
course:
Textbooks often use graphics to
aid the reader's comprehension.
Please examine the graphic.
Which step would you take first in
the process illustrated here?
7. • Stems should not contain negatives such as “except”
or “not.”
Not Acceptable:
Which of the following is not an example of a service company?
Better:
Which of the following is an example of a service company?
8. The following is also a negative stem:
Which of the following should be avoided when writing a topic sentence?
Including supporting details (correct)
Including a topic and an opinion
Providing an accurate explanation of what will follow
Including clear and easy to follow language
This question asks the student to choose something that’s incorrect, with
distractors that are “correct.”
We want to avoid this style of question because of a concept called imprinting.
We want our students to remember the correct answer. Think of it as positive reinforcement: students
should be receiving validation that what they’ve learned and concluded is right. In our curriculum and
our items, we want to model the correct answer for the students; negatives can inadvertently reinforce
the wrong concept.
9. Composing Keys and Distractors
• The second half of a question consists of a key and
three distractors.
• Distractors should be plausible, but incorrect.
What kind of computer skills will an employer
look for first in a job applicant’s resume?
Microsoft Office
World of Warcraft
Pokémon
Mario Brothers
What kind of computer skills will an employer
look for first in a job applicant’s resume?
Microsoft Office
QuickTime
Microsoft Silverlight
Google
Not plausible Plausible, but incorrect
10. One of the biggest problems item writers can run into is creating distractors that are good—
but not too good. In other words, the distractors have to be plausible without being logical
answers. Take a look at the following items. Can you see how a student could argue for
answering with one of the distractors?
Ex. 1: Team Development and Motivation
Tom is building a team. What characteristic elements best
define the team he eventually builds?
• A shared goal and an interdependent group of people
(correct)
• Independent team members and goal achievement
• Diversity and perseverance
• Working groups and customer service focus
Ex. 2: Fundamental Reading
Which of the following actions would you take before reading
an unfamiliar textbook?
• Preview the book and scan the chapter titles and sub-
headings (correct)
• Look up the author and determine her purpose for writing
the book
• Read the summary at the end of each chapter
• Look up all the unfamiliar words in a dictionary
When writing, try to look at the item from a student perspective. If you wouldn’t be able
to answer the question on a test, the student certainly won’t be able to. Your course
design team member will help you catch most distractor problems, but please do keep
this issue in mind.
11. • Keys and distractors will appear in a randomly generated order on
the test. Thus, answers such as “none of the above” and “all of the
above” are not appropriate.
• Multiple selection answers (for example, “Both a and b” or “Only b
and c”) will also not work on the platform and should be avoided.
More on Distractors
12. Structure
• Keys and distractors should match one another in
terms of length and grammatical structure.
This prevents students from basing their choices (consciously or
subconsciously) on minor structural differences
• They should also be capitalized and do not have a
period at the end
There is one exception: In a course such as Fundamental Writing, which
involves recognizing and correcting complete sentences and punctuation,
keys and distractors can take periods.
13. When practicing active listening, what does it
mean to paraphrase the speaker's message?
• To restate what you think the speaker was
trying to communicate
• To write down exactly what the speaker is
saying as he or she is talking
• To make bullet points about what the speaker
is communicating
• To remember only the factual information that
the speaker communicated
When practicing active listening, what does it
mean to paraphrase the speaker's message?
• To restate what you think the speaker was
trying to communicate
• Writing down what the speaker says
• To make bullet points about what the
speaker is communicating and follow up with
the speaker when he or she finishes
• Remember factual information from the
speaker’s message
Look at the examples above. The key and distractors in the left-hand
example do not match in terms of structure and length. A student may
use these differences to eliminate or choose answers. The example on
the right-hand side shows a corrected version of the question.
14. • Avoid giving clues in the stem that will lead the student to the right
answer. In other words, don’t use the same words in both the stem
and the key.
1. _______ refers to a kind of economic cooperation wherein member countries are free to
independently determine trade policies with nonmember nations.
a. Trade zone
b. Export processing zone
c. Free trade
d. Port trade
Better: What type of economic cooperation involves member countries independently
determining trade policies with nonmember nations?
15. Most of the information for a question should be in the stem, rather than in the
key or distractors. Avoid duplicating material in the answer choices, and include
the information in the question stem instead.
The European Union is an example of an economic union because:
a. The union frames the economic policies of the member countries individually
b. Some areas of the member countries have economic and other laws that are more free-
market-oriented than the country's national laws
c. The union frames common administrative policies for the member countries
d. Member countries are free to independently determine trade policies with nonmember
nations
Better:
What type of policies does the European Union frame for their member countries?
a. Economic
b. Free-market-oriented
c. Common administrative
d. Trade
(Removed redundant information from distractors and added to stem)
16. • Stems should ideally be between two and three sentences
long, and keys and distractors should consist of a word,
phrase, or sentence.
Students have a 90-minute time limit on their final exams, most of which will have 50 questions.
If students are spending more than a minute reading each question, they will have trouble
finishing the exam on time.
Another consideration is student fatigue. Because the tests are relatively long, some students
will begin to perform poorly as they become exhausted later in the test. Excessive numbers of
long stems or distractors will contribute to fatigue and frustration levels.
In some cases, a longer stem is acceptable. Take a look at the
examples of overly lengthy and acceptable stems, keys, and
distractors on the next slides.
Length
17. Longer stem is acceptable:
In this case, all of the information in the stem is necessary to understand the question itself (contrast
this to the stem from Conflict Management on the next slide, which didn’t require all of the dialog and
information to understand the question). These longer stems will be more common in courses such as
business or marketing, which often use case studies and realistic scenarios.
Principles of Marketing
After conducting extensive research on tablets, John bought an iPad from the Apple store. After using it
for a couple of weeks, he realized it does not have the functionality he assumed it would have. He is
wondering if he should have obtained more information and made more comparisons than he did
before purchasing his iPad. Which of the following is John suffering from?
Post-purchase dissonance (correct)
Product-evaluation psyche
Evaluative-criteria psyche
Post-purchase euphuism
18. Stem too long:
This stem is long and convoluted. Imagine being in the middle of a timed test and coming across this - it
could be stressful because you’d be uncertain of what the question was asking and what you should be
paying attention to. One possible fix would be to cut some or all of the dialog component.
Conflict Management
Andrew was talking to Jordan and said, "That teacher has it in for me. She always ignores me in class and
totally cheated me on that last assignment. I am not going to tolerate her anymore. I'm going to give her
a chance to make this right and apologize, but if not, I will have her job." So, Andrew's initial goals were
relational (being ignored) and content (being cheated). When he went into Mrs. Russell's office, he said,
"I would like you to regrade my essay response on the last test. I really don't think you understood what I
meant, Mrs. Russell". In response, she said, "Alright Andrew, I will look at it and return it to you in our
next class." He smiled and left her office. Changing goals during a conflict is common. What goal was he
smiling about when he left Mrs. Russell's office?
19. Answer and distractors too long:
One sign that you can shorten your distractors is repetition. If something within the answers/distractors is being repeated, try
moving that information to the stem. In the example, the first phrases (“Though the actors/actresses in pictures were
occasionally identified in the trade press” and “studios resisted publicizing their stars in the popular press in a deliberate
attempt”) could either be deleted or put in the stem.
Introduction to Film
In the early years of cinema (around 1900-1910), studios refrained from identifying
their actors/actresses in the media. Why did they do this?
A: Though the actors/actresses in pictures were occasionally identified in the trade press, studios resisted publicizing their stars
in the popular press in a deliberate attempt to prevent a costly star system
D: Though the actors/actresses in pictures were occasionally identified in the trade press, studios resisted publicizing their
stars in the popular press in a deliberate attempt to communicate specific personas crafted for each major actor/actress
D: Though the actors/actresses in pictures were occasionally identified in the trade press, studios resisted publicizing their
stars in the popular press in a deliberate attempt to prevent fans from intruding on the lives of stars
D: Though the actors/actresses in pictures were occasionally identified in the trade press, studios resisted publicizing their
stars in the popular press because the studios felt the movie was the star and not those who appeared in it; everyone who
worked on a film was equal
21. Many of our students are international or are returning for a degree after
many years of not being in school. Keep this in mind when composing
questions, and try to match the level of vocabulary to the level of the course
(the wording in a 100 level course will look quite a bit different from that in a
600 level course).
• Don’t use slang or idiomatic language, and be aware of cultural or socio-
economic issues in your choice of words (for example, not everyone will
know what foie gras is)
• Stay away from jargon and terminology unique to the textbook(s) of the
course
• Avoid bias and stereotyping, even when you’re presenting something as
incorrect information
• Include only abbreviations that are commonly used in discussing the
subject matter being tested (for example, m/s in Physics)
22. • Be creative: student engagement will impact performance
• Incorporate job-related situations, especially in higher level courses
• Give clues to a problem and ask for a best course of action
• Avoid having an excessive number of long or involved questions;
students have a time limit of 90 minutes on the final exam
• When in doubt, ask your course design team member!
Other Tips
