TESTA is Faculty of Arts and Social Sciences. Efficiency in Assessment and Feedback. Data-driven approach to fast-tracking quality assurance to make responsive changes to assessment.
TESTA at UNSW, Sean Brawley, TESTA Summit 16 Sept 2013
1. The UNSW experience
with TESTA
Faculty of Arts and Social Sciences
Professor Sean Brawley
Associate Dean (Education)
2.
3. UNSW Assessment Project Goals
1. Improve efficiency in the conduct of
assessment in order to reduce academic
workloads and ensure that student’s time on
tasks is not spent on unnecessary activities,
which have the sole purpose of generating a
grade.
2. Improve the educational effectiveness of
assessment tasks to ensure that they engage
and motivate students in their learning efforts.
4. 3. Increase the alignment of assessment tasks
with the disciplinary goals and the learning
outcomes of individual courses such that
student efforts are directed towards achieving
the key learning objectives.
4. Improve the alignment of assessment activity
and processes across the faculty.
5. Increase the congruence between Faculty
assessment processes and practices and the
University’s assessment policies, procedures and
aspirations.
5. Assessment Tasks in H&P
Final Exam
10%
Tut Particip/Prep
11%
Individual Present
3%
Tests/Quizzes
8%
Group Work
3%
Short writing
11%
Tut Writing
18%
Extended Writing
36%
7. UNSW HP assessments tasks against those of the University of
Sydney, University of Melbourne, University of Nottingham, George
Mason University and UCLA combined.
50
45
40
35
30
25
UNSW
Others
20
15
10
5
0
Extended Writing
Short Writing
Exams
Tut particip
Indiv Present
Group Work
Tests/Quizzes
9. Amount of words per level
6000
5000
4000
Level 1
3000
Level 2
Level 3
2000
1000
0
USYD
MELB
GM
NOTT
10. FASS Assessment Project
Faculty
Working
Party
Asses.
Tool
Prototype
Student
Time on
Task
Trial
Trial S1, 2012
15 courses
1000+ students
53 staff diaries
170 student diaries
Efficiency /
Quality
Assurance
Innovation /
Quality
Improvement
Faculty Assessment
Tool
TESTA
1700
Students
Complete
AEQ
60
Students
Focus
Groups
• Incorporates GA Tool
• Operational S2, 2012
• Aids governance,
• Mainstreams innovation
• Self-pop course guides
Reduce average student time on task from 114 to 85 hours
Establish staff marking expectations (time/words per student)
11.
12. Workshop Feedback
• “Interesting project that will enable innovation in
both efficiency and pedagogy”
• “The work shop was much more useful than I
thought it was going to be …the information
about TESTA has reassured me that there will be
a quality/pedagogy component to the
exercise, rather than only a resources/efficiency
drive. I now have a better idea how TESTA brings
the student voice to the fore, and think this is a
valuable undertaking”
13. Workshop Feedback
• “Thanks for the opportunity to do this workshop. Ive
learned much and realise how I can improive my own
practice and help steer colleagues in productive
directions”
• “The challenge is how to service implementation with
very limited resources and competing demands and
performance expectations … Positive education frame
of reference is a good place to start … with a proven
model of change. Thanks for a good experience in
collective problem solving”
14. Workshop Feedback
• “I fear that everyone knows that ultimately what
we decide will have very little impact on decisions
that are made higher up, and that [these
decisions] are not based on pedagogical
considerations. On that basis, its very frustrating
… The 2 days were really interesting for me. I do
think, however, that the major purpose for which
it ran, is unlikely to be achieved. It is difficult not
to be cynical about the pressures coming from
above — and they are not pedagogical”
15. Workshop Feedback
“Learnt heaps — so good to get an opportunity to
discuss assessment”
“TESTA sessions did not inspire enthusiasns when I
said I would attend but I have been pleasantly
surprised by how useful learning about the system
and process has been. I now want to revisit the
program I convene — this would be so useful for
anyone who developing a course or program”
16. Workshop Feedback
• “I thought the TESTA methodology was very
good and thorough but not realistic for the
time and resource constraints that UNSW
operates under”
18. “The teachers made it clear from the start what
they expected from students”
Became
“My lecturers/tutors made it clear from the start
what they expected from students”
19. “It was often hard to discover what was
expected of me in this course”
Became
“It was often hard to discover what was
expected of me in my major/program”
20. “I learnt new things while preparing for the
exams”
Became
“I learnt new things while preparing for the
exams/class tests”
22. Assessment Experience Questionnaire
• Slight changes to some of the 28 AEQ
questions
• Online instrument
• Explode email to nearly five thousand
undergraduate students
23.
24.
25. Assessment Experience Questionnaire
• Slight changes to some of the 28 AEQ
questions
• Online instrument
• Explode email to nearly five thousand
undergraduate students
• 1820 responded (37%)
27. Demographics
•
85% of students domestic, 15% international – would rep the average
•
Age
17-21 — 55%
22-30 — 37%
31 and over 6%
•
Gender
72% were female (higher than Faculty average)
•
Level
100 level — 28%
200 level — 22%
300 level — 27%
400 level above — 21% (greater than average)
•
GPA
pass average 16%
credit average 48%
a distinction average 31%
4% high distinction average
1% failing average
28. 1: I used the feedback I received to go back over what I had done in my work
1: I used the feedback I received to go back over what I had done in my work
2%
Strongly Disagree
2%
Strongly Disagree
7%
Disagree
7%
Disagree
21%
Neutral
21%
Neutral
56%
Agree
56%
Agree
15%
Strongly Agree
15%
Strongly Agree
0%
10%
0%
20%
10%
Strongly Agree
30%
20%
Strongly Agree
Agree
Agree
30%
Neutral
Neutral
40%
Disagree
Disagree
40%
Strongly Disagree
Strongly Disagree
50%
50%
60%
60%
29. 3: I received hardly any feedback on my work
5: The assessment system made it possible to be quite selec ve about what
parts of courses you studied
10%
Strongly Disagree
45%
Strongly Disagree
Disagree
3%
20%
Disagree
23%
Neutral
28%
Neutral
16%
Agree
43%
Agree
5%
Strongly Agree
6%
Strongly Agree
0%
5%
0%
10%
5%
15%
10%
Strongly Agree
20%
15%
Strongly Agree
Agree
Agree
25%
20%
Neutral
Neutral
30%
25%
Disagree
Disagree
35%
30%
40%
35%
Strongly Disagree
Strongly Disagree
45%
40%
50%
45%
30. 5: The assessment system made it possible to be quite selec ve about what
parts of courses you studied
5: The assessment system made it possible to be quite selec ve about what
parts of courses you studied
3%
Strongly Disagree
3%
Strongly Disagree
20%
Disagree
20%
Disagree
28%
Neutral
28%
Neutral
43%
Agree
43%
Agree
6%
Strongly Agree
6%
Strongly Agree
0%
5%
0%
10%
5%
15%
10%
Strongly Agree
20%
15%
Strongly Agree
Agree
Agree
20%
Neutral
Neutral
25%
25%
Disagree
Disagree
30%
30%
35%
35%
Strongly Disagree
Strongly Disagree
40%
40%
45%
45%
31. 11: It was possible to be quite strategic about which topics you could affo d
r
not to study
11: It was possible to be quite strategic about which topics you could affo d
r
not to study
3%
Strongly Disagree
3%
Strongly Disagree
25%
Disagree
25%
Disagree
34%
Neutral
34%
Neutral
35%
Agree
35%
Agree
3%
Strongly Agree
3%
Strongly Agree
0%
5%
0%
10%
5%
10%
Strongly Agree
Strongly Agree
15%
15%
Agree
Agree
20%
20%
Neutral
Neutral
25%
Disagree
Disagree
25%
30%
30%
Strongly Disagree
Strongly Disagree
35%
35%
40%
40%
32. 17: The way the assessment worked in this major/program meant you had
to study every topic
17: The way the assessment worked in this major/program meant you had
to study every topic
4%
Strongly Disagree
4%
Strongly Disagree 34%
Disagree
34%
Disagree
30%
Neutral
30%
Neutral
28%
Agree
28%
Agree
5%
Strongly Agree
5%
Strongly Agree
0%
5%
0%
10%
5%
10%
Strongly Agree
Strongly Agree
15%
Agree
Agree
15%
Neutral
Neutral
20%
20%
Disagree
Disagree
25%
25%
Strongly Disagree
Strongly Disagree
30%
30%
35%
35%
33. 7: It was always easy to know the standard of work expected
5: The assessment system made it possible to be quite selec ve about what
parts of courses you studied
6%
Strongly Disagree
22%
Strongly Disagree
Disagree
3%
20%
Disagree
25%
Neutral
28%
Neutral
41%
Agree
43%
Agree
6%
Strongly Agree
6%
Strongly Agree
0%
5%
0%
10%
5%
15%
10%
Strongly Agree
20%
15%
Strongly Agree
Agree
Agree
20%
Neutral
Neutral
25%
25%
Disagree
Disagree
30%
30%
35%
35%
Strongly Disagree
Strongly Disagree
40%
40%
45%
45%
34. 10: Thesacademic staff e emed more interested in tes ng what I had
memorised than what I understood
10: Thesacademic staff e emed more interested in tes ng what I had
memorised than what I understood
16%
Strongly Disagree
Strongly Disagree
Disagree
16%
46%
46%
Disagree
19%
Neutral
19%
Neutral
16%
Agree
16%
Agree
3%
Strongly Agree
3%
Strongly Agree
0%
5%
0%
10%
5%
15%
10%
15%
Strongly Agree
Strongly Agree
20%
20%
Agree
Agree
25%
25%
Neutral
Neutral
30%
30%
Disagree
Disagree
35%
35%
Strongly Disagree
Strongly Disagree
40%
40%
45%
45%
50%
50%
35. 16: Whatever feedback I received on my work came too late to be useful
16: Whatever feedback I received on my work came too late to be useful
6%
Strongly Disagree
6%
Strongly Disagree
32%
Disagree
32%
Disagree
29%
Neutral
29%
Neutral
25%
Agree
25%
Agree
7%
Strongly Agree
7%
Strongly Agree
0%
5%
0%
10%
5%
10%
Strongly Agree
Strongly Agree
15%
Agree
Agree
15%
Neutral
Neutral
20%
20%
Disagree
Disagree
25%
25%
Strongly Disagree
Strongly Disagree
30%
30%
35%
35%
36. 26: I learnt new things while preparing for the exams/class tests
26: I learnt new things while preparing for the exams/class tests
4%
Strongly Disagree
4%
Strongly Disagree
15%
Disagree
15%
Disagree
29%
Neutral
29%
Neutral
42%
Agree
42%
Agree
9%
Strongly Agree
9%
Strongly Agree
0%
5%
0%
10%
5%
15%
10%
Strongly Agree
20%
15%
Strongly Agree
Agree
Agree
20%
Neutral
Neutral
25%
25%
Disagree
Disagree
30%
30%
35%
35%
Strongly Disagree
Strongly Disagree
40%
40%
45%
45%
37. 27: I understood things be er as a result of the exams/class tests
27: I understood things be er as a result of the exams/class tests
6%
Strongly Disagree
6%
Strongly Disagree
18%
Disagree
18%
Disagree
36%
Neutral
36%
Neutral
33%
Agree
33%
Agree
7%
Strongly Agree
7%
Strongly Agree
0%
5%
0%
10%
5%
10%
Strongly Agree
Strongly Agree
15%
15%
Agree
Agree
20%
20%
Neutral
Neutral
25%
Disagree
Disagree
25%
30%
30%
Strongly Disagree
Strongly Disagree
35%
35%
40%
40%
38. 24: O en I found I had to study things without having a chance to really
understand them
24: O en I found I had to study things without having a chance to really
understand them
5%
Strongly Disagree
5%
Strongly Disagree 29%
Disagree
29%
Disagree
23%
Neutral
23%
Neutral
33%
Agree
33%
Agree
10%
Strongly Agree
10%
Strongly Agree
0%
5%
0%
10%
5%
10%
Strongly Agree
Strongly Agree
15%
Agree
Agree
15%
Neutral
Neutral
20%
20%
Disagree
Disagree
25%
25%
Strongly Disagree
Strongly Disagree
30%
30%
35%
35%
51. 4: You had to study the en re syllabus to do well in the assessment comparison of language major student responses to overall fig res
u
4: You had to study the en re syllabus to do well in the assessment comparison of language major student responses to overall fig res
u
1%
Strongly Disagree
5%
1%
12%
Strongly Disagree
Disagree
5%
33%
12%
Disagree
31% 33%
Neutral
28%
31%
Neutral
40%
Agree
27%
Agree
28%
40%
27%
15%
Strongly Agree
7%
Strongly Agree
15%
7%
0%
5%
0%
10%
5%
10%
15%
20%
15%
20%
Language major streams
Language major streams
25%
25%
Overall
Overall
30%
30%
35%
35%
40%
40%
45%
45%
52. 17: The way the assessment worked in this major/program meant you had
to study every topic - comparison of language major student responses to
overall figures
17: The way the assessment worked in this major/program meant you had
to study every topic - comparison of language major student responses to
overall figures
0%
Strongly Disagree
4%
0%
18%
Strongly Disagree
Disagree
4%
34%
18%
Disagree
28% 34%
Neutral
30%
28%
Neutral
41% 30%
Agree
28%
Agree
41%
28%
13%
Strongly Agree
5%
Strongly Agree
13%
5%
0%
5%
0%
10%
5%
10%
15%
20%
15%
20%
Language major streams
Language major streams
25%
25%
Overall
Overall
30%
30%
35%
35%
40%
40%
45%
45%
53. 18: To do well in the major/program all you really needed was a good
memory - comparison of language major student responses to overall
u
fig res
18: To do well in the major/program all you really needed was a good
memory - comparison of language major student responses to overall
u
fig res
13%
Strongly Disagree
19%
13%
33%
Strongly Disagree
Disagree
19%
41%
33%
Disagree
27% 41%
Neutral
21%
27%
Neutral
16%
Agree
14%
Agree
21%
16%
14%
11%
Strongly Agree
5%
Strongly Agree
11%
5%
0%
5%
0%
10%
5%
10%
15%
20%
15%
20%
Language major streams
Language major streams
25%
25%
Overall
Overall
30%
30%
35%
35%
40%
40%
45%
45%
54. 4: You had to study the en re syllabus to do well in the assessment comparison of Humani es student responses to overall fig res
u
5%
Strongly Disagree
4: You had to study the en re syllabus to do well in the assessment comparison of Humani es student responses to overall fig res
u
5%
5%
40%
Strongly Disagree
Disagree
5%
33%
40%
Disagree
29% 33%
Neutral
28%
29%
Neutral
21%
Agree
27%
Agree
28%
21%
27%
6%
Strongly Agree
7%
Strongly Agree
6%
7%
0%
5%
0%
10%
5%
10%
15%
20%
15%
Humani es
Humani es
20%
Overall
Overall
25%
25%
30%
30%
35%
35%
40%
40%
55. 25: Doing exams/class tests brought things together for me - comparison of
Humani es student responses to overall figr es
u
25: Doing exams/class tests brought things together for me - comparison of
Humani es student responses to overall figr es
u
11%
Strongly Disagree
8%
11%
33%
Strongly Disagree
Disagree
8%
21%
33%
Disagree
30% 21%
Neutral
37%
30%
Neutral
24%
Agree
31%
Agree
37%
24%
31%
1%
Strongly Agree
3%
Strongly Agree
1%
3%
0%
5%
0%
10%
5%
10%
15%
20%
15%
Humani es
Humani es
20%
Overall
Overall
25%
25%
30%
30%
35%
35%
40%
40%
Foundation member Group of Eight Research intensive52K student population85th Times Higher Ed Rankings52nd QS World Rankings
First stage – stocktaking and bencmarking
The broader projectHow we came to TESTA — Paul Hyland
Responbdents chance to win 10 shopping vouchers valued at $50 or an IPad
4913 undergrads, 999 postgrad coursework then HDR
32% of students from the BA30% of students provided additional comments — may be less than if we had done paper but probably got more responses85% of students domestic, 15% international – would rep the average55% 17-2137% 22-306% 31 and over72% were female (higher than Faculty average)28% 100 level22% 200 level27% 300 level12% 400 level (greater than average)9% 500 level (Eductaion combined degrees)16% pass average48% credit average31% a distinction average (not the case!)4% high distinction average (slightly under)1% failing averageArts students 32%
85% of students domestic, 15% international – would rep the average72% were female (higher than Faculty average) 400 level above — 21% (greater than average)a distinction average 31% (not the case!)4% high distinction average (slightly under)
Things I feared
Disappointed
disappointed
Surprised
pleased
Provoked reflection
Bit of a concern.
Statement 24 (‘Often I found I had to study things without having a chance to really understand them’) is concerning, with more students agreeing with this than disagreeing (43% compared to 34%). Even more Level 1 students agreed with this (50% strongly agree/agree with only 25% strongly disagree or disagree).
Good students did use feedback.
Students who do not achieve well in the program do not perceive they have received feedback on their work as much as those that do well. Is this due to differing feedback levels? Or with difficulties in appreciating/utilising the feedback they have been given?
Mature age students appear less strategic in their approach to assessment than do younger students. Particularly true for those older students who are in 100 level (see below).
International students also felt they needed to study the entire syllabus more than domestic students.
Mature age students felt the need to put the hours in regularly.
International students appear to have felt they got more out of exams than domestic students did.
Not much difference
Felt studying entire syllabus was very important, much more agreement than for students overall (4)
Also felt that assessments meant they had to study every topic, more important than for students overall (17)
Linked to this, more felt that a good memory was important than overall (18)
Did not feel entire syllabus needed to be studied as much as other students – more strategic learning perhaps (4)
Exams were less important in bringing coherence to the course than it was for other students (25)