It's not news that those of us in the testing world struggle with a variety of challenges. What's one of the largest? Test security - which is not likely to go away anytime soon. But, we also have measurement issues as our tests sometimes measure more than we want them to, and may not be as fair to certain subgroups in the population. In addition, costs to develop and administer tests have gone up with the use of technology, a fact opposite to how technology use has generally affected other industries. Finally, testing critics seem to be getting bolder in their claims, some of which are uncomfortably close to accurate. This webinar looks at how, through technology, changing the traditional multiple choice question can help substantially with these challenges and others. You won't want to miss this webinar, hosted by Caveon CEO David Foster. Dr. Foster will present information from his recently published white paper Compelling Test Design Technology: Discrete Option Multiple Choice.

1. Caveon Webinar Series:
Solving Big Problems
with DOMC, a 21st
Century Technology
April 23, 2014
Dr. David Foster,
CEO
Caveon Test Security

2. Agenda for Today
• What is DOMC?
• DOMC effects on testing
problems
• Creating DOMC items
• SUPER DOMC
• Challenges with DOMC
2

3. Major Problems We Face
Security is abysmal
Quality of measurement can be better
Costs are too high
Every test is unfair to someone
3

4. DOMC can help…
What is DOMC?
Discrete Option Multiple Choice
Simplest Description: Options for the multiple choice
question are presented randomly one at a time with
YES and NO buttons.
Options stop appearing when the test taker
answers the question correctly or incorrectly.
4

5. DOMC Examples
The best way to understand DOMC is to try it
out…going to trydomc.com
We are going to look at some varieties:
• simple multiple choice
• multiple choice with more than one correct answer
• multiple choice scored correct with responses to
distractors only
• DOMC with larger option pools (SUPER DOMC)
5

6. How can DOMC
help solve these
problems?
6

7. Major Problem #1:
Tests Are Stolen and Used to Cheat
• Tests are stolen routinely and shared quickly on the
Internet.
• New technologies (e.g., Google Glass) will be very
effective.
• Cheating by using pre-knowledge of questions is
impossible to detect by proctors.
• Old fashioned cheating remains a threat.
• Security problems directly affect the test score and
society’s trust in those scores.
7

8. Security Solution: How Does DOMC Help?
• DOMC exposes much less of the item content
• Makes it more difficult to steal accurate
questions
– SCIENTIFIC RESEARCH:
• 2.5 options on the average are presented compared to
4 or 5 for the traditional MC item.
• About a 50% reduction in the exposure of critical
content.
– Absolutely NO MEASUREMENT LOSS
– SUPER DOMC reduces the exposure more
dramatically (more on this later)
8

9. How Does Reduced Exposure
Improve Test Security?
• You can’t capture what you can’t see!
• You can’t share what you didn’t see!
• If you want to cheat, you can’t depend on what
others tell you about the test!
9

10. RESEARCH: College Students
10

11. Expert Opinion
“DOMC testing may also considerably reduce
the danger of unwanted copying and sharing of
test items because many answer options do not
have to be presented at all.”
Sonja Willing, Martin Ostapczuk, and Jochen Musch, University of
Dusseldorf, in Advances in Health Science Education, 2014
11

12. Major Problem #2:
Quality of a score from multiple
choice tests is less than it should be
What would happen to the quality
of a cake if you added 10% of
some totally unrelated
ingredient? Chili Peppers?
Vinegar? Asphalt?
• Traditional multiple choice questions are like
this cake.
12

13. Major Problem #2:
Quality of a score from multiple
choice tests is less than it should be
• SCIENTIFIC RESEARCH: TESTWISENESS
adds about 10% to 15% to a person’s score.
• This is a large portion of a score (and
time/money) devoted to measuring an
irrelevant skill!
• It is CRITICAL that we remove as much
TESTWISENESS as possible if we want to
more accurately measure what is important.
13

14. Measurement Solution:
How Does DOMC Help?
• SCIENTIFIC RESEARCH:
– DOMC reduces TESTWISENESS effect
• Test takers cannot compare options, a major
tactic of TESTWISENESS
• Use of TESTWISENESS is also a FAIRNESS
issue. (more on this later)
14

15. Windfall
Test takers soon realize that “being prepared”
means learning the subject matter of the test, not
learning how to game the question format.
15

16. RESEARCH: Student Reactions
College students
prefer the
traditional MC
format…
…because
DOMC removes
test taking cues
Of course, they would also prefer not taking
tests at all, not going to the dentist, and not
having their ID’s checked.
16

17. Major Problem #3:
Tests cost too much to create and deliver
 Programs create large
numbers of items for
security reasons (to foil
thieves).
 The cost to administer a
computerized test is higher
than it should be, in my
opinion.
This may not be a major concern to most programs,
but who wouldn’t like more cash to spend?
17

18. Cost Solution:
How Can DOMC Help?
In at least two ways:
1. DOMC test questions remain in service,
uncompromised, for much longer, perhaps
indefinitely. Test development costs are cut
significantly.
1. SCIENTIFIC RESEARCH: Testing seat time
is reduced by 10% to 39%.
1. Less seat time means less money paid for
test administration.
2. Opportunity for greater testing volumes.
18

19. Major Problem #4:
Fairness
FAIRNESS Facts:
 The testing industry is getting hammered by
critics—justifiably so in some cases
1. We allow TESTWISENESS in multiple choice tests
2. Our multiple choice tests are unfair to some sub-
groups
 These are, in my opinion, accurate criticisms
 The most NOBLE thing we can do is make our
tests as fair as possible.
19

20. How Does DOMC Help with Unfairness?
Removes Some Inequality of Test Prep
A major goal of test prep courses is to teach tactics
to best answer multiple choice questions
(TESTWISENESS).
Test preparation courses costs money and can’t be
afforded by many test takers, who are
disadvantaged as a result.
DOMC removes the need to
take test prep course to
learn test taking skills. 20

21. How Else Is DOMC More Fair?
It Is Simple to Do!
DOMC format is cognitively
simpler making it easier for
all, including:
– young children
– older adults
– those with test anxiety
– those where the language
of the test is not their
native language
– those where the culture
does not include MC tests
– many others
21

22. Creating DOMC Items
• Item banking and test administration systems
need to be modified for DOMC, just as they
need to be changed for any new item type.
• Kryterion has supported a simple version of
DOMC.
• Excelsoft has built the more complete version
of DOMC.
• Here is a screen from their system, SARAS.
22

23. SARAS Test Development
Screen #1: Stem
23

24. SARAS Test Development
Screen #2: Options
24

25. SARAS Test Development
Screen #3: Scoring/Stopping Rules
25

26. SUPER DOMC Item
Definition: A SUPER DOMC item has more correct and
incorrect options than would be normal for the
traditional version.
For example, having 3 correct and 7 incorrect
options would enhance the advantages described
in these slides.
It is not always easy to do to add options,
but almost always possible.
26

27. Example: SUPER DOMC Item
Objective: Add two 2-digit numbers.
MC Stem: Is this addition correct?
Correct Options look like this: 12 + 79 = 91
There are hundreds of possible correct options
Incorrect Options look like this: 33 + 17 = 40
There are thousands of possible incorrect options
27

28. Challenges with DOMC
• Most test takers don’t like it.
• Takes only a little bit of practice.
• Item banking systems and test administration
systems have to be modified.
• Test administration must use a “forward-only”
design (e.g., no re-viewing items).
28

29. To Learn More…
There is no charge to the licensee to try out
DOMC or for doing research.
Caveon will provide you will the help you
need to get started.
Link to DOMC white paper –
http://www.caveon.com/white-papers/
If questions, email
david.foster@caveon.com
29

30. Questions?
30

31. Thank you!
Dave Foster, CEO
Caveon Test
Security
Follow Caveon on twitter @caveon
Check out our blog www.caveon.com/blog
LinkedIn Group “Caveon Test Security”
31