The document discusses operational processes, technology, and support for online assessment based on discussions from the Online Assessment Special Interest Group. It identifies several key challenges faced by institutions in transitioning to online assessment, including how to organize operational processes, adapt face-to-face assessments, deploy technology, and support staff and students. Technology was widely deployed using both commercial and in-house solutions for proctoring, plagiarism detection, and assessment delivery. However, technological solutions also increased student anxiety so support was crucial. Case studies highlight the importance of cross-institutional teams, flexibility, training, and guidance to help navigate these challenges of online assessment.

1. Designing Online Assessment
Operational Processes, Technology
and Support
Carlton Wood - The Open University UK
EADTU empower webinar week, 16 June 2022

2. Online Assessment Special Interest
Group
• Establishment of SIG January 2021; First
meeting March 2021; 5 meetings in 2021-
2022
• 18 members institutions from 14 countries
• 37 HE practitioners + 18 contributors of
good practice
• 15 Good Practice case studies
• 4 Focus areas
• Assessment Futures
• Assessment Design
• Trust, Privacy and Ethics
• Operational processes, technology and
support
https://online-assessment.eadtu.eu https://doi.org/10.5281/zenodo.6563226

3. Assessment Future is already present
• Online Assessment is here to
stay
• Transition phase from exam
hall to online home exams
• Distance Teaching sector
long history of home-based
assessment
• EADTU member institutions
best placed to lead
• Great variety of hybrid
solutions to online
assessment

4. Operational processes, technology
and support
• Must start with thanks to all the contributors to the SIG
• Without their generous inputs of time, thoughts and practises then there
would not be anything to report
• This presentation will cover Chapter 3 and I understand that fellow authors
have already spoken in this empower session
• All inputs were gathered as previously described – either from Case Studies
provided from SIG members or via Padlet contributions
• Distillation of common themes
• Aim to highlight key ideas / practises that could be carried forward

5. Operational processes, technology
and support (continued)
The over-arching challenges that were identified that were faced by all
institutions were:
• How did institutions organise themselves to deliver the required operational
processes?
• Could face-to-face provision be adapted for on-line delivery?
• To what extent was technology deployed to assist in the delivery of
operational process?
• How did Universities support staff and students whilst delivering the process?

6. 1) How did institutions organise
themselves to deliver the required
operational processes?
• Rapid onset of the pandemic required urgent rapid responses
• General uneasiness expressed which suggested that participants in the SIG
somehow felt that they may be an outlier “do universities have already
frameworks and guidelines for online examinations?”
• No organisation had all the answers
• Little evidence provided that anyone had generated a previously prepared
process that it could swiftly bring out and deploy
case study from Open University UK

7. 1) How did institutions organise
themselves to deliver the required
operational processes? ….cont.
• The previous Good Practice identified 6 reasons or themes as to why this
particular model appeared to work well. These themes were:
• common purpose,
• urgency,
• listening and being heard,
• student partnerships,
• professionalising communication
• cross university membership
• Similar themes were presented from most of the members although there were
obviously nuances dependent upon individual circumstances
• Tension “creating meaningful assessments, measuring high thinking levels,
which do not take up too much faculty time to grade and give feedback”

8. 2) Could face-to-face provision be
adapted for on-line delivery?
• It was clear that many institutions had good examples where assessment was
delivered well in a face-to-face environment
• The challenge was to what extent could the existing assessment be adapted
to make it deliverable and useful in an on-line environment
case study from Universidad Técnica Particular de Loja (UTPL) in Ecuador

9. 2) Could face-to-face provision be
adapted for on-line delivery?
• Some evidence of identifying individuals had experience of on-line delivery
and so bringing these individuals together
• Within the same University different approaches were required. Despite the
University level overviews that were set up in the majority of institutions to
handle process, it was clear that a ‘one size fits all’ was never going to work
• Degree of flex so individual academic areas did not have to follow a single set
unified process.
• Sometimes these decisions were left to faculties “large scale proctoring took
place in one faculty, open book exams in some of the other faculties”
• Interaction between the academic side of Universities and Information
Technology (IT) section of the same institution
• “education development department sets out a framework, checklist, it goes
through IT-department for working protocols to see if that's all fine”

10. 3) To what extent was technology
deployed to assist in the delivery
of operational process?
To bring process change, application and deployment
of technology featured heavily. Five themes emerged
• Sheer range of solutions that were employed
• Solutions developed in house versus those available commercially
• Deployment of technology –proctoring and plagiarism tools
• Changes that were introduced could increase concern/anxiety amongst
students
• Difficulties that were encountered with technology
• Use of technology in different ways – “use of Zoom as a tool to support the
student and visual supervision of the process in a synchronous way;
Through this, I can verify the identity of the student” and secondly “oral
examinations are replaced by online "zoom" sessions”

11. 3) To what extent was technology
deployed to assist in the delivery
of operational process? (cont)
• Clear that respondents were happy to utilise commercially available software
whilst others either used “in-house” solutions that was already developed
• Some took the time to (rapidly) develop new additional in-house solutions -
“development of a completely new application for secure and reliable online
exams, AvEx, through the collaborative work of experts in technology and
methodology”
case study from Universidad Técnica Particular De Loja | Ecuador

12. 3) To what extent was technology
deployed to assist in the delivery
of operational process? (cont)
• One development that demonstrated a wider perspective than just a single
university was reported from Finland and relies on a national based approach
to developing a technologically based solution “a national platform for e-
exams is in use and being developed further”
• This example is not an isolated one either and both the TeSLA project
mentioned in the introduction to the report and the development and sharing
of software solutions is highly encouraging
• Technologies for proctoring or for detecting plagiarism by students – strongly
linked to the trust element described in a previous session
• All the technological solutions had potential to cause students stress -
students need to be carefully guided in the 'technical' process to reduce
stress. Provide an opportunity for a 'try out' [is also helpful] – [and] give clear
instructions and also feedback on what is happening during the process”

13. 4) How did Universities support
staff and students whilst
delivering the process?
• A major challenge that all institutions faced during the pandemic was how
best to support both staff and students in a period of extensive change
• Delivering this support in a non face-to-face manner added another level of
complexity
• This Case Study from the Open University of the University of Jyväskylä in
Finland highlights key aspects
case study from Open University of the University of Jyväskylä in Finland

14. 4) How did Universities support
staff and students whilst
delivering the process? (cont.)
• Some of the support that
Universities were offering was
reactive but at other points
Universities required students to
take steps themselves to ensure
that on-line assessments stood
more chance of taking place
successfully
• Some institutions generated
support that was equally
applicable to both students and
staff
• Examples included providing
technical support via telephone
during live examination periods

15. Lessons learned going forwards
• Timescales
- if time is short do much of what is described in the report
- if more time is available then identifying what tools and processes
would best fit your university would be helpful
• How prepared are the staff for any proposed changes?
• Have a variety of support mechanisms to offer and to make available
(UniDistance Vademecum example)
• “Lot of supportive documents, protocols proposals, GDPR instructions, use
cases presented by experienced teachers to other teachers; still in many
ways teachers struggled”
• We should question whether taking what we do in a face to face environment
and simply converting that to on-line delivery is the most effective approach, it
did in some circumstances and not in others
• “We have built a pilot project to evaluate pedagogical challenges with online
exams. This will lead to a report and guidelines and final strategy and
regulations for next academic year”

16. Questions?
carlton.wood@open.ac.uk