Presentatie gegeven op de Educause 2011, door Alfred Hartoog en Nynke Bos

1. Bringing e-assessment
to a higher maturity on
a national scale
dr. A. Hartoog, SURF, Utrecht
drs. N.R. Bos, Fontys, Eindhoven
Educause october 2011

2. Program
introduction
1 problems, questions, all somehow 10 min
related to e-assessment
discussion
2 how is this situation in the US and/or 15 min
Canada?
elaboration
3 how are we tackling these issues in the 15 min
Netherlands, on a national scale
4 questions/answers 10 min
1
Educause october 2011

3. context
The Netherlands:
- surface area 40.000 km2 (US is 270 times bigger)
- population density: 491 people/km2 / (US: 34 people/km2)
- about 60 universities,
- about 600.000 students
- two handfuls of government funded and commercial
R&D institutes
2
Educause october 2011

4. Problems, questions…(1)
- many students drop out in their first year of study
- and if they don’t, their success rate is less than optimal
- can we improve this situation via (digital) testing?
3
Educause october 2011

5. Problems, questions…(2)
- Assessment is a key proces in education
- Testing students properly is tedious and time-
consuming
- Teachers in general are no trained specialists in
the assessment proces, and are usually already
overloaded
- How to improve this situation?
4
Educause october 2011

6. Problems, questions…(3)
- Clinical reasoning is a key competence for doctors
- proper assessment in their education is crucial
- given the high number of students in medical
schools,
- how on earth do you assess this competency
reliably
5
Educause october 2011

7. Problems, questions…(4)
- teaching statistics is a key component in many
educations
- statistics is often a stumbling block for students
- statistics typically has slow and fast students
- how do you make statistics education, training and
assessment more effective and fun for the slow and
fast alike
6
Educause october 2011

8. Problems, questions…(5)
- apprentices in nursing experience a gap between
education and practise, leading to drop-outs
- there are many educations in nursing, with the
associated variety in end-terms
- how to tackle this?
7
Educause october 2011

9. Discussion
- can e-assessment reduce teacher workload?
- does information technology raise the quality of testing?
- can e-assessment reduce students dropping out?
- is it possible to test complex skills with closed questions?
- how could e-assessment improve statistics education?
- could e-assessment reduce the reality gap in nursing?
- would a national infrastructure for e-assessment be useful?
- how can you stimulate collaboration between educations?
- how do you introduce e-assessment in law education?
all the above on a national scale!
8
Educause october 2011

10. So, what we did?
Context:
Role of SURF in Higher Education in the Netherlands
Program:
Testing and test driven learning
Impact:
Things that work, things that don’t
9
Educause october 2011

11. SURF: ICT innovation by and for
higher education and research
Research universities, universities of
applied sciences, and research institutions
work together in SURF in order to:
• improve the quality of research and
education by:
• creating pioneering ICT innovations.
For more than twenty years,
collaboration within SURF has led
to products and services that the
participating institutions could not
have achieved on their own.
10
Educause october 2011

12. SURF is a partnership
between all Dutch institutions for higher education & research
42
Universities of
applied sciences
14
National Library
Universities
Smaller
Collaboration Netherlands
Organisation for
institutions of
higher education
= Scientific
Research (NWO)
Success factor! Royal Netherlands
Commercial Academy of Arts
R&D institutes and Sciences
Dutch Organisation (KNAW)
for Applied Scientific
Research (TNO),
Telematics Institute
11
Educause october 2011

13. SURF as an innovator
Collaboration within SURF makes it possible to:
• carry out a joint (public/private) survey
of what higher education & research
requires/will receive
• cluster ICT requirements on the demand
side (demand pull)
• arrive at sustainable ICT services suitable
for joint use
• be innovative in contracting out services
• encourage the use of innovative services
(supply push).
Educause october 2011 12

14. SURF subjects
- advanced networks
- security/identity management/federation
- collaborative environments
- software procurement/licence management
- digital rights & open access
- open educational resources
- SURF academy
- information architecture
- national information brokerage
13
Educause october 2011

15. The SURF method
Implementation
SURF directs and coordinates the
implementation of funded projects:
• Projects are carried out close to the
source of expertise: at institutions of
higher education and in enterprise.
• SURF sets up the programme, invites
tenders, ensures independent quality
assessment and monitoring, makes
project results available.
Educause october 2011 14

16. Program “Testing and
test driven learning”
Problems in higher education in the Netherlands:
- high drop-out rate
- high workload on teachers
SURF launched a national program to tackle these
problems with e-assessment
15
Educause october 2011

17. Purpose of this program
- improving transition from secondary higher education with
diagnostic tests and revision material, to be jointly
developed by institutions,
- improving students’ progress by means of progress tests,
- reducing the time needed to develop tests by
collaboration, i.e. developing collections of digital tests,
- reducing the time taken to mark tests by deploying
computer-based testing,
- providing expertise and professional development for
instructors in the field of testing.
16
Educause october 2011

18. programme composition
the program focusses on
three main area’s:
1. enabling institutions to jointly
develop computer-based tests and
assessments
2. realising an expertise network,
aimed at improving quality of testing
3. realising a national technical
infrastructure to combine successful
national initiatives for computer
based testing
17
Educause october 2011

19. What kind of projects?
bachelor nursing:
- “body of knowledge”
medicine:
- interuniversity progress test medicine (2)
- testing of clinical reasoning
- testing with 3D images (research)
math and statistics
- foundation of national test database
- research into effects of progresstesting
- test-based learning (adaptive testing) (2)
teacher education
- research into testing abilities of teacher educators
- research into methods for portfolio assessments
-> and: new tender by 1 november
18
Educause october 2011

20. Math & Statistics projects
- Stumbling block
- Faster and slower students
- Based on earlier project “Mathematics Garden”
-> computer adaptive monitoring
- Students: can practice
can adjust their learning process
- Reduction in drop out
- Teachers: can adjust their teaching process
- Reduction in workload
University of Amsterdam, University of Utrecht, Twente University

21. Medical projects
- Low dropout rates students but high workload
teachers (doctors)
- Development of questions on a national scale
- Sharing these in national databases
- Different goals:
- Measuring progress
- Measuring Clinical Reasoning (complex skills)
- Improvement gap practice/ theory
- Involvement: all nursing schools, all medical schools

22. Research projects
3D Testing (CT/MRI)
The project aims at improving the psychometric
quality and validity of assessment of 3D images.
Also gain insight into the cognitive processes
required in the interpretation of 3D images
Progress testing and study success
- First year economic students will get weekly
progress tests
- Will this increase [retention rates]?
- Does this also effect workload for teachers, costs of
assessment, quality of testing?

23. Network of expertise
Fysically:
- a Special Interest Group of inspired specialist
- seminars, schools, master classes (SURF Academy)
- small scale practical research projects
Virtually:
web portal that acts as a centre of expertise:
- good practises
- white papers
- links to literature
- ask and find specialists
- blogs, discussions forum
- agenda
free accessible and maintained by the higher education community
22
Educause october 2011

24. National infrastructure
ambition:
investigate, realize and exploit the advantages
of a common infrastructure for computer based
assessment on a national scale
healthy in all respects, to stand on its own when
the program finishes, and external funding is
discontinued
23
Educause october 2011

25. what’s to gain? (1)
sharing systems:
B
user i/f
A application
user i/f s combi
application
C
middleware user i/f
s user i/f application
middleware application
?
s
operating s
operating systems & middleware middleware
systems & networks operating operating
networks systems & systems &
networks networks
hardware hardware
hardware hardware
24
Educause october 2011

26. what’s to gain? (2)
sharing system management:
B
helpdesk
A funct.mgnt combi
helpdesk C
helpdesk
funct.mgnt appl.mgnt helpdesk
?
funct.mgnt funct.mgnt
appl.mgnt
appl.mgnt
technical appl.mgnt
technical mgnt technical
mgnt technical mgnt
mgnt
infra.mgnt infra.mgnt infra.mgnt infra.mgnt
25
Educause october 2011

27. lessons learned up to now
- subject domains in education
(math, health, law, humaniora, etc) are very
different
- educational institutions are very different
- the advantages of sharing an infrastructure are not
always obvious
- computer based testing is far from mature
26
Educause october 2011

28. so, what do we do
- confine within domains
- investigate and explore along the functional lines
- make advantages explicit
- join already successful initiatives
- take small steps, but think big
- develop business models
and this might actually work:
- example
27
Educause october 2011

29. collaboration in medical
and nursing education
BoKS Clinical
Body of Knowledge Reasoning
& Skills
education in
nursing and ?
medicine
?
Testing
iPTM ? in 3D
inter University
Progress Test Medicine AdaPT
28
Educause october 2011

30. Questions, answers
shoot!
Educause october 2011 29

31. Alfred Hartoog
hartoog@surf.nl
www.surf.nl
Nynke Bos
n.bos@fontys.nl
www.amc.nl
www.fontys.nl
30
Educause october 2011

32. Impact,
how to measure
- this four-year program is one year underway
- high level goals:
- reduce number of students that drop-out
- reduce workload on teaching staff
- raise quality level of e-assessment
How do you measure this?
basic problem: there are many factors (not related to
the program) that may have a dominant influence on
above subjects.
31
Educause october 2011

33. evaluation and impact
- decompose high-level program goals to manageable
project proportions
- find ways to measure these parameters, and assess
them during the project
- develop a model to arrange the findings in levels of
evidence (based on the knowledge-pyramid from Kennisnet)
- inspiration (this is great: it should work)
- perception (this actually works for me)
- existence (others confirm that it works)
- evidence (scientific proof that it works)
32
Educause october 2011

34. Evidence:
measured benefits
Perception:
experienced benefits
Existence:
deployment of the project
Inspiration:
Idea
33
Educause october 2011