The presentation reflects on systematic and continuous quality schemes in large-scale, online learning environments. It also focus on the experiences with quality barometers and other quality enhancement tools and services at NKI Nettstudier –Scandinavia’s largest online education provider.

1. Quality schemes in cooperative
learning environments
Morten Flate Paulsen
Professor of Online Education
A 50-minutes presentation at the
International Scientific Conference on ICT and Education
Montreal, May 2012
http://twitter.com/MFPaulsen www.slideshare.net/MortenFP
http://nettstudier.blogspot.com www.facebook.com/mfpaulsen
http://home.nki.no/morten www.eden-online.org/blog/ 1

2. Abstract
The presentation will reflect on systematic and
continuous quality schemes in large-scale,
online learning environments.
It will focus on the experiences with quality
barometers and other quality enhancement tools
and services at NKI Nettstudier –
Scandinavia’s largest online education provider.
2

3. About me
• Worked with online education since 1986
• Professor of Online Education at NKI Nettstudier , Norway
• President of European Distance and E-learning Network (EDEN)
• Doctor of Education from the Pennsylvania State University,
working for the American Center for the Study of Distance
Education (1990-1992)
• Adjunct Professor Athabasca University (1999 to 2005)
• Professor associado Universidade Aberta, Portugal (2009-2010)
3

4. www.eden-online.org
• 20th anniversary in 2011, 8 in secretariat, 11 in EC
• EDEN members from 400 institutions in 56 countries:
– 20th anniversary
– 200 institutional members
– 2000 potential members of NAP –
Network of Academics and Professionals
• Conferences
– Open Classroom conference ,Athens October 2011
– Annual conference, Porto June 2012
– Research Workshop, Leuven October 2012
• EU-projects with 168 institutions
• European Journal of Open, Distance and E-learning
www.eurodl.org

5. EDEN-members in Canada
• Athabasca University
• Canadian Network for Innovation in Education – CNIE
• Société de Formation a distance des Commissions Scolairs du Que
5

6. Annual Conferences
21. Oporto 2012
Open Classroom 20. Dublin, 2011
9. Athens, 2011 19. Valencia, 2010
8. Oporto, 2009 18. Gdansk, 2009
7. Stockholm, 2007 17. Lisbon, 2008
6. Poitiers, 2005 16. Naples, 2007
5. Copenhagen, 2002 15. Vienna, 2006
4. Barcelona, 2000 14. Helsinki, 2005
3. Balatonfüred, 1999 13. Budapest, 2004
2. Crete, 1997 12. Rhodes, 2003
1. Oslo, 1995 11. Granada, 2002
10. Stockholm, 2001
Research Workshops
8. Moscow, 1999
7. Leuven, 2012 7. Bologna, 1998
6. Budapest, 2010 6. Budapest, 1997
5. Paris, 2008 5. Poitiers, 1996
4. Barcelona, 2006 4. Birmingham, 1995
3. Oldenburg, 2004 3. Tallinn, 1994
2. Hildesheim, 2002 2. Berlin, 1993
1. Prague, 2000 1. Krakow, 1992 6

7. Planning for Oslo
EDEN’s Annual Conference, June 2013

8. November 2011
NKI Nettstudier was established as a
correspondence school in 1910
• One of Europe’s largest providers of online distance education
• About 150 000 enrolments in online courses since 1987
• Broad range of subjects from secondary to master level
• About 400 distance education courses online (all in Norwegian)
• About 12 000 online students in about 40 countries
• About 70 % women
• Individual progress plans/individual follow-up
• Thousands of potential learning partners
• Global student catalogue with 1200 profiles
• Always room for more students
• Exams at local schools and embassies
8
www.nki.no

9. College, vocational- and
secondary school
Picture from: http://youtu.be/NXBNVauG9bE
9

10. 400 online courses
Picture from: http://youtu.be/NXBNVauG9bE
10

11. Focus on individual progress plans

12. Example of individual progress plan
12

13. NKI: A pioneer in Online Education
1986: First LMS – specifically developed for distance learning
1987: First online distance education course
1992: Mandatory course for new NKI tutors on the Internet
1996: First web-based courses
2002: First m-learning courses
2003: Speech synthesis in all courses
2004: Individual planning system and individual follow up
according to the student’s plan
2004: Continuous tracking of turn-around time
2006: Learning partner system
2008: Global student profiles
2010: Peer counselling system for tutors
2011: Quality barometer for tutors and course materials
13

14. The most interesting pedagogical challenge in our
lifetime is to provide online education that
combines individual freedom with meaningful
cooperation.
I have struggled with this challenge since I first
introduced my Theory of Cooperative Freedom in 1992.
www.ednews.org/articles/10626/1/An-Interview-with-Morten-Flate-Paulsen-Focusing-on-His-T
14

15. Cooperative Freedom is discussed in my book
Online Education which is available from
http://issuu.com/mfpaulsen
15

16. More resources on Cooperative Freedom
• Cooperative online education
www.youtube.com/user/nki2008#p/a/u/2/HVXtbLg5ycE
• Online education barometers
www.youtube.com/watch?v=gyoe7dGIwkE&feature=relmfu
• Transparency in Cooperative Online Education
www.irrodl.org/index.php/irrodl/article/view/671/1267
• E-viva em Portugal http://nettstudier.blogspot.com/
• Cooperative Online Education
http://seminar.net/index.php/volume-4-issue-2-2008-previousissuesmeny-124/100-cooperative-online-ed
• Cooperative Freedom and Transparency in Online Education
www.slideshare.net/MortenFP/futuralia-3400339
• Visualizing student profiles
www.slideshare.net/MortenFP/visualizing-student-profiles-through-nkis-online-catalogue-and-student-ne
• Transparency for Cooperation
http://toonlet.com/archive?m=s&i=11245

17. Six Dimensions of Freedom
Compilation of resources made by my students at
Universidade Aberta in Portugal
It is difficult to combine individual flexibility and cooperation
http://ppel4.pbworks.com/

18. One may say that:
Individual learning is conducted alone
Cooperative learning takes place in networks
Connectivism?
Collaborative learning depends on groups
18

19. NKI provides cooperative online education
Independence with the help of others
19

20. Cooperative learning
• Cooperative learning seeks to develop virtual learning
environments that allow students to have optimal
individual freedom within online learning communities.
• Cooperative learning is based on voluntary participation
in a learning community
• Cooperative learning benefits from transparency
NKI’s Philosophy: We facilitate individual freedom within a
learning community in which online students serve as
mutual resources without being dependent on each other.
20

21. Tools and strategies to enhance quality
in a virtual learning environment with:
• 12.500 students
• 400 courses
• 150 part-time teachers
• 60 full-time employees
should be systematic, continuous and cost-effective
www.nki.no 21

22. Strategies and tools to enhance quality
1. Large scale online education needs systematic,
continuous and cost-effective quality measures
2. Transparency improves quality
3. Teacher Support Services
4. Quality Barometers
22

23. Transparency
or
privacy?
23

24. 24

25. Transparency improves quality
…of system data, course content, course work etc….
25

26. Audio Journalism: Berghs School of Communication (Sweden)
Video Journalism: UPDATE – Danish School of Media and Journalism
Text Journalism: NKI Distance Education (Norway)

27. Transparency promotes cooperation
Dalsgaard, C. and Paulsen, M.F. (2009) Transparency in Cooperative O
IRRODL, Vol 10, No 3. 27

28. Three degrees of transparency
• Low: Private e-mail
• Medium: Class forum
• High: Global web2.0 services
Transparency provides better quality and more
cooperation, but both students and teachers
should be comfortable with the chosen
transparency level.
28

29. Online teachers deserve decent support services
http://nettstudier.blogspot.com/2011/08/online-teachers-deserve-decent-support.html
1992: Online course for online teachers 2010: Peer counseling and self-evaluation
2004: Response Barometer 2011: Quality Barometer for Teachers 29

30. Response Barometer for Teachers
• Introduced in May 2004
• Resulted in much discussion in the
teachers’ online forum
• A few teachers voiced strong criticism,
doubts and reservations
The average response time for all In 2008 NKI introduced
150 teachers in 2010 was 1.77 days. an automatic e-mail
reminder to the tutors
who had one ore more
assignments waiting for 7
days or more.
30

31. Barometer Questions
1. What to measure?
2. Which scale?
3. Transparent for whom?
4. Possible to show trends?
5. What about Evaluation Moment, Frequency and
Fatigue?
31

32. Online Education Barometers
should have individual readings
cooperative readings, aims,, warning levels
and transparent feedback loops
Ideal for Individual Start
and Progression
Transparent feedback to the right people 32

33. NKI’s Transparent:
Quality Barometer
for Online Teachers

34. Evaluation Icon
As soon as a teacher provides submission feedback and enters a grade, the
far right evaluation icon shows up in the planning tool at the student’s
start page.
Click the box to open it.

35. Evaluation Button
The student has to evaluate the teacher to embed the grade.
Click the evaluation button to open the teacher evaluation window.

36. Teacher Evaluation Window
Choose the teacher satisfaction level.
The options from left to right are:
Very dissatisfied, Dissatisfied,
Neutral, Satisfied and Very Satisfied.
Then enter some optional comments
on the teacher’s work.
The chosen satisfaction level
influences the Teacher Quality
Barometer as soon as it is
submitted.
After a random time delay for
anonymity reasons, the teachers can
see the comments in their Details
Report.

37. Teacher Quality Barometer
Percentages shown in the teacher’s individual barometer and the
cooperative barometer for all 140 teachers, are updated continuously as
additional evaluations are submitted.
The impressive barometer readings show the percentages of Very satisfied
and Satisfied evaluations.
After a random anonymity delay, the student’s comments are shown in the
Teacher Details Report.

38. Teacher Details Report
Example of detailed quality report for a teacher including anonymous verbal comments.
38

39. NKI’s transparent Teacher Quality Barometer was
launched in 2011 for further quality improvements at
Scandinavia’s leading online education provider.
On September 7, NKI’s 12.000 online students started evaluating their
teachers.
On October12, NKI’s 140 online teachers could access the online
evaluation results.
At the launch, 2000 evaluations were accrued and 88% were
either Very satisfied or Satisfied.

40. Barometer Reading after 4017 Evaluations
2011Tutor quality barometer
40

41. Peer Evaluation of Teachers
In 2010, NKI introduced an online course structure for systematic peer evaluation
among all the teachers. Random feedback the teachers have given on student
submissions are sent to a colleague who evaluate it and give advice on possible
improvements.
Professor Torstein Rekkedal has conducted a survey among the first participants and
concluded that they perceive it as motivating and informative. They also reported that
they learned at least as much from giving feedback as they did from receiving feedback.
They will therefore both give and receive feedback on each other's work in the future.
41

42. Teacher Self-evaluation
Last school year, the majority of the teachers also
delivered self-evaluation of one of the feedbacks
they gave on student assignments. The experiences
were so positive that future self-evaluation will be
included in an online course structure in which the
teachers will use the online submission system to
evaluate their personal presentations, activity in
forums and feedback on student submissions.
42

43. Star teachers deserve Golden Apples along
with other stars who are awarded with gold
medals, Oscars and Grammies. Therefore,
NKI has introduced a new, and hopefully
sustainable, scheme for nominating and
awarding NKI-teachers of the year.
43

44. Thank you
I care, so I share:
http://nettstudier.blogspot.com
http://home.nki.no/morten
https://twitter.com/MFPaulsen
www.slideshare.net/MortenFP
www.facebook.com/mfpaulsen
www.eden-online.org/blog
44