2. • Advantage and limitations
• Enumerate the methods of objective evaluation
• Types of multiple choice question
• Advantage and disadvantage of MCQ
• The component of MCQ item
• Guideline for construction of an MCQ
• Item analysis
Objective
4. Advantage
• Objective in nature
• In depth evaluation of student !
• Easy to evaluate
• To test the candidate at the desired level of
difficulty
• Good for formative evaluation and for
screening a large number of examinees
• Standard of scoring can be kept constant for
many years
5. Disadvantage
• Poorly framed MCQ may be misleading
• Tendency among teacher to test trivial and
less important data
• Questions are difficult and time consuming to
construct
• Not very useful for summative evaluation
6. Types
• The single or one best response
• Multiple completion type
• Relationship analysis type
• Multiple true false completion type
• Matching type
• Case history type
• Pictorial/Graphical /Tabulated type
• Comparison type
7.
8. Testing Item Content Levels
• Recall – testing knowledge of isolated facts
• Interpret* – asking to review and reach some
conclusion
• Problem solve* – asking to take some action after
reading a situation
*Higher order thinking skills.
9. Writing for Different Cognitive Levels
• Recall
– Which of the following is the process of mitosis?
• Interpret
– Which of the following can be classified as a
meiosis error?
• Problem-solving
– A maternal blood sample revealed decreased
levels of AFP and estriol and elevated hCG. What
would be the next logical test to perform?
10. Purposes…
• Motivate students
• Identify areas of deficiency or further learning
• Determine final grades or make promotion
decisions
• Identify areas where the course/curriculum is
weak
11. What Should Be Tested?
• Exam content should match course/clerkship
objectives
• Important topics should be weighted more
heavily
• The testing time devoted to each topic should
reflect the relative importance of the topic
13. Direction for framing MCQs
• Design each item to measure an important
learning outcome
• Present single, compact, clearly formulated
stem
• Use simple and clear language
• Stem should be a statement , not a word
• State the stem in positive form as far as possible
• The word ‘NOT’ and ‘EXCEPT’ should be
underlined
14. Direction …cont
• Grammar ,Style, length, plausible distractors
• Avoid clue that suggest answers, similar length
and form
• ‘All of the above’, ‘none of the above’
• Item with numerical alternatives , arrange
them in rank order
• Testwiseness
• Irrelevant Difficulty
15. Issues Related to Testwiseness
• Grammatical cues
• Logical cues
• Absolute terms
• Long correct answer
• Word repeats
• Convergence strategy
• B or C position overused
16. The forbidden fruit that Eve offered Adam was
an:
(a) apple (b) pear
(c) banana (d) melon
(e) carrot
17. Keep the alternatives simple
• When your body adapts to your exercise load,
a. you should decrease the load slightly.
b. you should increase the load slightly.*
c. you should change the kind of exercise you are doing.
d. you should stop exercising
• When your body adapts to your exercise
load, you should
• a. decrease the load slightly.
b. increase the load slightly.*
c. change the kind of exercise.
d. stop exercising
18. Implausible distractors:
• In a study of the effect of diet on risk of diabetes,
the researcher can manipulate a number of
variables including the amount of food,
carbohydrates, proteins or fats consumed. During
the experiment the amount of food, protein and
fat subjects consumed remained the same. Only
the amount of carbohydrates consumed
changed. What was the independent variable in
this study?
a. amount of food consumed
b. amount of carbohydrates consumed
c. amount of protein consumed
d. amount of fat consumed
19. Plausible distractors:
• In a study of the effect of diet on risk of diabetes, the
researcher measured how likely the subjects were to
get diabetes and how severe their symptoms were if
they developed the disease. To prevent amount of
exercise from influencing the results, the researcher
held it constant in the two groups he was studying.
What was the independent variable in this study?
a. likelihood of developing diabetes
b. severity of symptoms
c. diet
d. amount of exercise
dietary pattern
20. Possibility of a “Random Pass”
Depends on the number
of answer options per question
and the number of questions!
22. Adjustment for Guessing
Negative Marking…
– Elimination strategy reduces odds of wrong
answer penalty
– Subtracting a percentage of the number of wrong
answers obtained from the final grade
– Give a grade of 4 for a correct answer and a score
of -1 for a wrong answer on a 4 choice question
23. Item analysis
• Difficulty index
• Discrimination index
• Distractor effectiveness
• To create a question bank
• Used in classroom teaching to modify teaching content
or methodology
24. Purpose of Item Analysis
–Evaluates the quality of each item
–Rationale: the quality of items
determines the quality of test (i.e.,
reliability & validity)
– May suggest ways of improving the
measurement of a test
25. • Difficulty index
– The item difficulty for item i, pi , is defined as the
proportion of examinees who get that item correct.
• Method for Dichotomously Scored Item
• Method for Polytomously Scored Item
• Grouping Method
N
R
P
maxX
X
P
2
LU PP
P
26. Item Difficulty (cont.)
Interpreting the p-value...
example:
100 people take a test
15 got question 1 right
What is the p-value?
Is this an easy or hard item?
27. Item Difficulty (cont.)
Interpreting the p-value...
example:
100 people take a test
70 got question 1 right
What is the p-value?
Is this an easy or hard item?
28. 0.5 + 10
Too EasyToo Difficult
Level of Difficulty
Index Range Difficulty Level
0.00-0.20 Very Difficult
0.21-0.40 Difficult
0.41-0.60 Average/Moderately
Difficult
0.61-0.80 Easy
0.81-1.00 Very Easy
29. ITEM DISCRIMINATION
• The extent to which an item
differentiates people on the
behavior that the test is
designed to assess.
• the computed difference
between the percentage of
high achievers and the
percentage of low achievers
who got the item right.
30. Index of Discrimination
• (used for dichotomously scored items)
D = PH - PL
• We need to set one or two cutting scores to divide the
examinees into upper scoring group and lower
scoring group.
• PH is the proportion in the upper group who answer
the item correctly and PL is the proportion in the
lower group who answer the item correctly.
• Values of D may range from -1.00 to 1.00.
31. Example
There are 140 students attending a world history
test. If 18 examinees in upper group answer
item 5 correctly, and 6 examinees in lower
group answer it correctly, then calculate the
discrimination index for item 5.
32. Example 2
50 Examinees’Test Data on 8-Item Scale About Job
Stress.
Item 1 2 3 4 5 6 7 8
PH
PL
.54 .81 .47 .32 .51 .18 .63 .56
.32 .56 .11 .05 .10 . 23 .25 .19
D .18 .25 .36 .27 .41 -.05 .38 .37
33. Guidelines for Interpretation of D Value
• D≥.40, the item is functioning quite satisfactorily
• .30≤ D≤.39, little or no revision is required
• .20 ≤ D≤.29, the item is marginal and needs revision
• D≤.19, the item should be eliminated or completely
revised
0 + 1- 1
Good
Discrimination
Reverse
Discrimination
34. What’s a “Good” Distracter?
A good distracter is a plausible answer:
• CM = common student mistakes, a recurrent
misconception about topic
• TG = too general option, that is too broad to
answer the question
• TS = too specific option, option focuses on one
detail in stem
• OT = off topic, option misses the answer
completely
35. Guidelines for constructing a MCQ paper
• Include a sufficient number of items to have
validity
• Time allotted
• Group according to format
• Instructions for negative marking
• Place easy questions first, easy moderate and
difficult questions in sufficient numbers
• All part of an item should on same page
37. Use of machine !
• Computer
• OMR
Go green, Think green, Act green !
http://www.go-green.ae/
38. Further reading…
• http://www.teambasedlearning.org/
• Constructing Written Test Questions For the
Basic and Clinical Sciences Third Edition
(Revised)
• Multiple Choice Questions Not Considered
Harmful
by
Karyn Woodford, Peter Bancroft accessed from
http://crpit.com/confpapers/CRPITV42Woodford.pdf on 09 Aug14
Notas del editor
What is an Implausible Distractor?
Plausible: reasonable, appearing worthy of belief
Implausible: provoking disbelief
An implausible Distractor can be defined as an answer choice that can be quickly
eliminated by someone with little or no knowledge of the topic being tested. For example,
an answer choice could be considered implausible if it:
is composed of relatively unimportant information along with other choices
that focus on more important concepts (trivial Distractor); or
does not grammatically follow from the stem (specific determiners); or
contains words such as “always” or “never” which suggest an incorrect
option.
Student answers based on test-taking skills like Grammatical cues Logical cues Absolute terms Long correct answer Repeating word Convergence strategy
Student has difficulty answering due to Long, tricky and complicated stems Inconsistent data Unparallel language in options Illogical order of options “None of the Above” option Answer hinged to another item Imprecise terms
Issues Related to Testwiseness
• Grammatical cues - one or more distractors don’t follow grammatically from the stem
• Logical cues - a subset of the options is collectively exhaustive
• Absolute terms - terms such as “always” or “never” are in some options
• Long correct answer - correct answer is longer, more specific, or more complete than other options
• Word repeats - a word or phrase is included in the stem and in the correct answer
• Convergence strategy - the correct answer includes the most elements in common with the other options
Issues Related to Irrelevant Difficulty
• Options are long, complicated, or double
• Numeric data are not stated consistently
• Terms in the options are vague (eg, “rarely,” “usually”)
• Language in the options is not parallel
• Options are in a nonlogical order
• “None of the above” is used as an option
• Stems are tricky or unnecessarily complicated
• The answer to an item is “hinged” to the answer of a related item