A talk at the conference of higher education in Tallinn, Jan 2013

1. Visualizing Student
Feedback:
a case for mixed
quant-qual approach
Margus Niitsoo, Kaspar Kruup
University of Tartu

2. This talk
● Quantitative feedback – Why?
● Visualization? What for?
● Feedback for curriculum level?

3. My perspective
● as a student
– graduated just 2 years ago
● as a lecturer
– 5 years teaching experience
● as the curriculum manager
– past 2 years
● as an engineer
– MSc and PhD in Computer Science

4. Qualitative feedback
● “Stricter deadlines would have
disciplined us to study more!”

5. Qualitative feedback
● “Stricter deadlines would have
disciplined us to study more!”
● “It was hard to hear the lecturer”

6. Qualitative feedback
● “Stricter deadlines would have
disciplined us to study more!”
● “It was hard to hear the lecturer”
● “The course sucked!”

7. Qualitative feedback
“The lecturer was one of the best I
have seen in my studies”
vs
“The lecturer was hard to follow,
monotonous, inept and boring”
Which is (more) true?

8. Quantitative feedback
The averaged student
rating for the course was:
4.2

9. Quantitative feedback
The averaged student
rating for the course was:
4.2 / 5

10. Quantitative feedback
The averaged student
rating for the course was:
4.2 / 5
1 2 3 4 5

11. Quantitative feedback
The averaged student
rating for the course was:
4.2 / 5
1 2 3 4 5
Respondents: 14/100

12. University of Ontario

13. University of Amsterdam
All in all, I give this class the following grade:
(1=very bad, 5=excellent)

14. An often unanswerable question
Does the score of 4.2
mean that I am
above average?

15. Add the averages (UT):
Institute avg | Faculty avg | Uni. avg

16. Next problem
Does the score of 4.2
mean that I am
above average?
By how much?

17. Easy!
The institute average was
4.0 with SD 0.3

18. Easy?
The average for courses
was 4.0 with SD 0.3
What (the hell) does
SD mean?

19. Standard deviation
Course avg: 4.2
Avg. for courses: 4.0
SD for courses: 0.3

20. Standard deviation
Course avg: 4.2
Avg. for courses: 4.0
SD for courses: 0.3
Quick calculation:
(4.2-4.0)/0.3 = 0.66

21. Standard deviation
Course avg: 4.2
Avg. for courses: 4.0
SD for courses: 0.3
Quick calculation:
(4.2-4.0)/0.3 = 0.66
Translation:
Better than ~70%

22. We are bad with numbers!
● Analysis of numbers is slow
● It is hard (requires effort)
● We are prone to mistakes!

23. We are bad with numbers!
● Analysis of numbers is slow
● It is hard (requires effort)
● We are prone to mistakes!
Also: a picture is worth
a thousand words!
(We can represent more with less)

24. The UT solution

25. The UT solution
(University of Amsterdam)

26. The UT solution

27. The UT solution

28. The UT solution

29. The UT solution

30. Pros
● Good quick overview
– With details also available, if needed
● Better information
– Ranking and distribution implicit
● Emphasizes important
– Comparison with mean and other courses
● Increased interest in feedback
– More visually attractive and accessible than numbers

31. Some research:
● On-line survey among students
– 107 respondents
– 4 tasks of feedback analysis + opinion survey
– 3 groups for each task
● Tables with numbers, Simplified graph, Full graph
● Main findings
– Graphs took less time to process in analysis tasks
– Most preferred graphs over numbers
(but some had a strong preference for tables with numbers so both needed)
– Workload question result often misinterpreted
(since the “best” response for that question is 0 instead of 2)
(this is joint work with Kaspar)

32. Curriculum management
● Course planning
– What to teach when
– Who teaches what
● Quality assurance
– Identify bright and problem spots
– Spread good practices
– Aid lecturers with problem spots

33. Good overview essential!
● Course planning
– What to teach when
– Who teaches what
● Quality assurance
– Identify bright and problem spots
– Spread good practices
– Aid lecturers with problem spots

34. Standard overview (UT)

35. Problems (as before)
● Absolute values
– Is 3.2 good or bad?
● Just one value
– Leads to oversimplification
● Mostly numbers
– Hard to get an overview

36. Proposed solution
(Inspired by heatmaps used in Bioinformatics)

37. Summary
● Quantitative feedback has a use
– Finding the bright spots and problem areas
● Presentation of data is important
– Although sadly this is often neglected

38. Further work
● Qualitative research into the
perceptions of the academic staff
of the new system
– Currently in progress with a larger team
● Good review of systems currently
used throughout the world
– Does your uni. have something better?

39. Thank you!
Any questions and
comments would be
welcome!
(If you have any later, try Margus.Niitsoo@ut.ee)