A presentation at IACAT 2012 Sydney

1. Psychological aspects of CAT:
seeking item selection rules which
do not decrease test takers’ learning
self-efficacy and motivation
Tetsuo KIMURA *1 *2 & Keizo NAGAOKA *2
*1 Niigata Seiryo University
*2 Graduate School of Human Science, Waseda University
IACAT Conference 2012 (Sydney, Australia)
Aug. 12-14, 2012

2. 1
CAT for Everyone
MOTTO
&
Happy CAT

3. 2
Outline
• Previous Study (Kimura & Nagaoka, 2011)
▫ How test-takers feel about CAT
• Current Study UCAT
▫ Research Question
▫ Designe
▫ Result
▫ Conclusion

4. 3
Previous Study: Questionnaire Research
(Kimura & Nagaoka, 2011)
Participants: 156 Japanese university freshmen
One week after they had taken an English CAT (CASEC)
1. Did you feel the CAT easy or difficult?
Just right, 13
Very A little A little easy,1
difficult, 82 difficult, 60
Very easy, 0
0% 20% 40% 60% 80% 100%

5. 4
Previous Study: Questionnaire Research
(Kimura & Nagaoka, 2011)
Participants: 156 Japanese university freshmen
One week after they had taken an English CAT (CASEC)
2. Did you feel discouraged or unsatisfied with the
experience?
Dicouraged, Neutral, 35 Satisfied, 26
95
0% 20% 40% 60% 80% 100%

6. 5
Previous Study: Questionnaire Research
(Kimura & Nagaoka, 2011)
Participants: 156 Japanese university freshmen
One week after they had taken an English CAT (CASEC)
3. How much do you . . . ?
100
80 76
66 70
60
% 45
40 34
20
0
High School CASEC Satisfied Expected Disapointed

7. 6
Students reaction to CAT(Kimura & Nagaoka, 2011)
More difficult I could not
It was so
than entrance answer with
difficult 
exam  confidence 
Hana Yoko
Taro
I may not be
I dislike English Worst score I’ve
able to reach the
worse  ever got 
passing grade 

8. 7
Students reaction to CAT(Kimura & Nagaoka, 2011)
Every students tend to get 50% correct in CAT:
because such items maximize test information
for each student. However, they feel 34% correct.
Hana Yoko
Taro
False Low Advanced
beginner Intermediate Intermediate

9. 8
Students reaction to CAT(Kimura & Nagaoka, 2011)
School Exams 60-80%
40-60%
80-100%
Hana Yoko
Taro
Entrance Exams
60

10. 9
Hypothesis(Kimura & Nagaoka, 2011)
CAT tends to be very difficult to most of the students.
Experience of taking CAT discourages students.
Backwash effects Loss of self-confidence
Loss of self-efficacy
Loss of motivation

11. 10
Conclusion (Kimura & Nagaoka, 2011)
Most of the students (90%) feel CAT difficult.
Many of them (60%) feel discourage/unsatisfied with
the experience.
More attention may have to be paid to the
psychological effects of CAT.
Item selection rule to maximize information should
be changed even if it makes the test a little longer.

12. 11
Current Study: Research Question
Will manipulation of item difficulty in item
selection rules change students reactions to CAT?
Minimum number of CAT Items
Administered
Targetin S.E. (logits)
gP 0.5 0.4 0.3 0.2 0.1
0.5 16 25 45 100 499 CAT A
0.6 17 27 47 105 517
0.7 20 30 53 120 477 CAT B
0.8 25 40 70 157 625
0.9 45 70 125 278 1112 Linacre (2006)

13. 12
Current Study: Design
About 200 Japanese university freshmen took both CAT A
& CAT B with one month interval. They also answered
questionnaire immediately after each CAT
Targeting Number Expected
P of Item S.E.
Vocabulary &
0.5 16 0.5
Grammar
CAT A
Listening
0.5 16 0.5
Comprehension
Vocabulary &
0.7 20 0.5
Grammar
CAT B
Listening
0.7 20 0.5
Comprehension

14. 13
Current Study: Result
1. Did you feel the CAT easy or difficult?
Very A little Just right,
CAT B
difficult, 36% difficult, 49% 13% A little easy,
1%
Very A little Just right,
CAT A
difficult, 49% difficult, 45% 6%
A little easy,
0%
0% 20% 40% 60% 80% 100%
χ2 = 11.8 p = 0.008

15. 14
Current Study: Result
2. Did you feel discouraged or unsatisfied with the
experience?
Very discouraged, 9% A littele satisfied,
19%
CAT B Neutral, 51%
A littele Very satisfied,
discouraged, 14% 8%
Very discouraged, 13% A littele satisfied,
10%
CAT A A littele
Neutral, 47%
Very satisfied,
discouraged, 22% 7%
0% 20% 40% 60% 80% 100%

16. 15
Current Study: Result
2. Did you feel discouraged or unsatisfied with the
experience?
Very discouraged, 9% A littele satisfied,
CAT B 23% A littele
Neutral, 51% 27%
19%
Very satisfied,
discouraged, 14% 8%
Very discouraged, 13% A littele satisfied,
10%
CAT A
35% A littele
discouraged, 22%
Neutral, 47%
17%
Very satisfied,
7%
0% 20% 40% 60% 80% 100%
χ2 = 12.4 p = 0.014

17. 16
Current Study: Result
3. How much do you . . . ?
100
Previous
90
CAT A
80
CAT B
70
60
50
40
30
20
10
0
High School CAT Satisfied Expected Disapointed

18. 17
Current Study Conclusion
Changing the targeting P of item selection from 0.5 to
0.7, with 20% increase in the number of item
Less test-takers feel the CAT very difficult
More test-takers feel the CAT just right
Less test-takers feel discouraged with the experience
More test-takers feel satisfied with the experience
This item selection rule less decreases test takers’
learning self-efficacy and motivation

19. 18
Current Study: Result
3. How much do you . . . ?
100
Previous
90
CAT A
80
CAT B
70
60
50
40
30
20
10
0
High School CAT Satisfied Expected Disapointed

20. 19
Thank you for listening.
Tetsuo Kimura
(tetsuo.kmr@gmail.com)
Acknowledgements:
A part of the present study was supported by a Grant-in-
Aid for Scientific Research for 2010-2012 (No. 22520590)
from the Japan Society for the Promotion of Science.