2. Multiple Choice
Item Writing Workshop
Doug Mauger EdD
School Of Instructor Education
dmauger@vcc.ca
Nursing Workshop 2/12/2013
3. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Validity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
4. Introductions
• Name:
• What topics do you teach?
• How long have you been teaching?
• One thing you know about Item writing
• What do you want to get out of the workshop?
Nursing Workshop 2/12/2013
5. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
6. Put each
rider on
the top
of a
horse
Nursing Workshop 2/12/2013
7. Put each
rider on
the top
of a
horse
Nursing Workshop 2/12/2013
8. Put each
rider on
the top
of a
horse
Nursing Workshop 2/12/2013
9. Put each
rider on
the top
of a
horse
Nursing Workshop 2/12/2013
10. Put each
rider on
the top
of a
horse
Nursing Workshop 2/12/2013
14. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Validity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
16. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Validity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
18. Points to Ponder
There were flaws in 46% of the total
questions, with the number of flaws per
test ranging from 36% to 65%. In a
sample of examinations distributed to
first and second year medical students
(Downing, 2005)
Downing, S. M., 2005. The effects of violating standard item writing principles on tests and students: the con- sequences of using flawed test items on
achievement ex- aminations in medical education. Advances in Health Science Education 10 (2), 133–143.
Nursing Workshop 2/12/2013
28. 7. Process Validity (students not able to provide best
performance)
8. Content Validity (different instructor’s test reflects
different instruction)
9. Construct Validity (improper form of assessment)
10. Content Validity/Alignment (instruction and
assessment do not match)
Nursing Workshop 2/12/2013
29. Why is Validity and
Reliability important
to you as a
professional?
Nursing Workshop 2/12/2013
30. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Validity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
31. Alignment
Content and Assessment
Alignment
Are Critical
For Valid
Assessment
Nursing Workshop 2/12/2013
46. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
51. Remembering/Understanding Level
Describes the circulation through the heart
Defines the term systole
Nursing Workshop 2/12/2013
52. Application Level
Application (AP)
theories in providing care to clients (e.g., a
Apply the concepts of aging in
developing interventions for the
elderly
Nursing Workshop 2/12/2013
53. Critical Thinking Level revised
Blooms Taxonomy
Develop a plan for delivering services to elderly persons who
are homebound
Evaluate nursing research based on predetermined criteria
Analyzes the organizational structure of the community health
agency and its impact on client services.
Nursing Workshop 2/12/2013
55. Four Knowledge Dimensions
Factual Knowledge
The basic elements students must know to be acquainted with a discipline or
solve problems in it.
Conceptual Knowledge
The interrelationships among the basic elements within a larger structure
that enable them to function together.
Procedural Knowledge
How to do something, methods of inquiry, and criteria for using skills,
algorithms, techniques, and methods.
Meta-Cognitive Knowledge
Knowledge of cognition in general as well as awareness and knowledge of
one's own cognition
Nursing Workshop 2/12/2013
59. Writing a MC Item
What is the biggest
challenge for you in sorting
outcomes?
Nursing Workshop 2/12/2013
60. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
68. Points to Ponder Item
Constructing a MC
The present study examined the effect of NOTA on multiple choice
questions in a large classroom setting. Specifically, we
manipulated the place- ment of NOTA across versions of the test
such that, for each of 20 experimental questions, NOTA was either
not present, replaced the key, or re- placed the high, medium, or
low frequency dis- tractors respectively. We found that NOTA sig-
nificantly increased the difficulty of multiple choice questions only
when it was used as the key. We found no effect of NOTA replacing
any of the dis- tractors, regardless of their effectiveness. Addi-
tionally, we found no significant effect of NOTA on discrimination
scores, regardless of its place- ment.
2010 Matthew V. Pachaia, David DiBattistab, Joseph: The effect of ’None of the Above’ on multiple choice questions in a first year classroom A Preprint
submitted to Advances in Health Science Education
Nursing Workshop 2/12/2013
73. Varieties of MC Items
1. Single Correct Answer
2. Multiple Response
3. Combined Response
4. Multiple Response
Nursing Workshop 2/12/2013
74. Writing a MC Item
1. Design each item to measure an important learning
outcome
2. Present a single clearly formulated problem in the
stem of the item
3. State the stem of the item in simple, clear language
4. State the stem in positive form wherever possible
5. Word the alternatives clearly and concisely
6. Keep the alternatives mutually exclusive
Nursing Workshop 2/12/2013
75. Writing a MC Item
7. Keep the alternatives homogeneous in content
8. Keep the alternatives free from clues as to which
response is correct
9. Avoid the alternatives AOTA and NOTA
10. Use as many functional distractors as are feasible.
11. Include one and only one correct or clearly best
answer for each item
Nursing Workshop 2/12/2013
76. Writing a MC Item
Writing a MC Item
12. Present the answer in each of the alternative
positions approximately an equal number of
times, in a random order
13. Layout the items in a clear and consistent manner
14. Use proper grammar, punctuation, and spelling
15. Avoid using unnecessarily difficult vocabulary
16. Analyze the effectiveness of each item after each
administration of the test.
Nursing Workshop 2/12/2013
77. Rules Broken
in the
Content Free Test
Nursing Workshop 2/12/2013
78. Writing a MC Item
Item writing……
Nursing Workshop 2/12/2013
79. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
85. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
86. Writing a MC Item
What
So what
Now what
Nursing Workshop 2/12/2013
88. Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
• Valdity and Reliability
• Alignment
• Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop
Nursing Workshop 2/12/2013
89. Writing a MC Item
Thank You
Nursing Workshop 2/12/2013
90. ReferencesKrathwohl, David R. (2001). A Taxonomy for
Anderson, Lorin W. &
Learning, Teaching and Assessing: a Revision of Bloom’s Taxonomy.
New York. Longman Publishing.
http://shpliteracy.pbworks.com/w/file/fetch/54533674/NEW%20Blooms%20Taxono
my%20Question%20Stems.pdf
Nursing Workshop 2/12/2013
Notas del editor
No need to fold the paperNo need to cut or rip the paper
No need to fold the paperNo need to cut or rip the paper
No need to fold the paperNo need to cut or rip the paper
No need to fold the paperNo need to cut or rip the paper
No need to fold the paperNo need to cut or rip the paper
Knowledge/Comprehension (K/C)This level refers to the ability to recall previously learned material and to understand its meaning. It includes such cognitive abilities as knowing and understanding definitions, facts, principles and interpreting data (e.g., knowing the effects of certain drugs, interpreting data appearing on a client’s chart).
Application (AP)This level refers to the ability to apply knowledge and learning to new or practical situations. It includes applying rules, methods, principles and nursing theories in providing care to clients (e.g., applying principles of therapeutic communication to the nursing care of clients).In this question, the examinee would be required to use the cognitive ability of Application. That is, the examinee would be required to apply, from among the general measures to counteract pyrexia, those that are appropriate at this particular stage (e.g., onset). The examinee would also be required to apply knowledge of measures that should be taken initially given that an elderly person can become debilitated very rapidly from the effects of a fever.
Critical Thinking (CT)The third level of the taxonomy deals with higher-level thinking processes. It includes the ability to judge the relevance of data, to deal with abstractions and to solve nursing problems (e.g., identifying priorities of care, evaluating the effectiveness of nursing actions). The nurse should be able to identify cause-and-effect relationships, distinguish between relevant and irrelevant data, formulate valid conclusions and make judgments concerning the needs of clients.In this question, the examinee would be required to use the cognitive ability of Critical Thinking. That is, the examinee would be required to solve a nursing situation while considering the relationship between the client’s age and manifestations that have not been directly observed by the practical nurse. The examinee would have to make a judgment about the seriousness of this situation (e.g., possibility of meningitis) and determine the most appropriate intervention.
Knowledge/Comprehension (K/C)This level refers to the ability to recall previously learned material and to understand its meaning. It includes such cognitive abilities as knowing and understanding definitions, facts, principles and interpreting data (e.g., knowing the effects of certain drugs, interpreting data appearing on a client’s chart).
Application (AP)This level refers to the ability to apply knowledge and learning to new or practical situations. It includes applying rules, methods, principles and nursing theories in providing care to clients (e.g., applying principles of therapeutic communication to the nursing care of clients).In this question, the examinee would be required to use the cognitive ability of Application. That is, the examinee would be required to apply, from among the general measures to counteract pyrexia, those that are appropriate at this particular stage (e.g., onset). The examinee would also be required to apply knowledge of measures that should be taken initially given that an elderly person can become debilitated very rapidly from the effects of a fever.
Critical Thinking (CT)The third level of the taxonomy deals with higher-level thinking processes. It includes the ability to judge the relevance of data, to deal with abstractions and to solve nursing problems (e.g., identifying priorities of care, evaluating the effectiveness of nursing actions). The nurse should be able to identify cause-and-effect relationships, distinguish between relevant and irrelevant data, formulate valid conclusions and make judgments concerning the needs of clients.In this question, the examinee would be required to use the cognitive ability of Critical Thinking. That is, the examinee would be required to solve a nursing situation while considering the relationship between the client’s age and manifestations that have not been directly observed by the practical nurse. The examinee would have to make a judgment about the seriousness of this situation (e.g., possibility of meningitis) and determine the most appropriate intervention.