This presentation, given at the Assessment Tomorrow Vocational Assessment event in Leeds (17th Jan 2013) looks at experiences and lessons from supporting the delivery of e-assessment in the workplace to medical students at the University of Leeds.

1. Inspired assessment and learning technology
Lessons from Leeds:
5 ways to innovate vocational
assessment
Dave Waller
Product & Design Manager, MyKnowledgeMap Ltd.
http://www.flickr.com/photos/davehamster/6813051009/
©2013 MyKnowledgeMap Ltd 1

2. Inspired assessment and learning technology
Background
• We’re an assessment and learning technology
company based in York
• We have a strong history of supporting further
education, higher education and work-based
learners
• Our software systems are multi-award winning and
offer unique opportunities for learning, assessment
and evidence gathering
• We’ve spent the last 5 years working closely with
the University of Leeds on pioneering mobile
assessment projects
©2013 MyKnowledgeMap Ltd 2

3. Inspired assessment and learning technology
What are the University of Leeds doing?
• The School of Medicine have embedded mobile
assessment within their undergraduate medical
degree
• In years 4 and 5 students spend almost all of their
time on placement - in hospitals or other primary care
settings
• Face to face tutor-student contact is limited yet
assessment and feedback still need to happen
• Capture of evidence and proof of competence is key
to the medical curriculum
©2013 MyKnowledgeMap Ltd 3

4. Inspired assessment and learning technology
• From September 2010, students have been
equipped with iPhones and a suite of resource
apps, e-books and assessment tools
• Tutors create assessments, tasks and forms
which can be sent directly to student phones
• Students complete their assessments in the
workplace, either on or offline, and submit their
responses for review
• Tutors review responses online and provide
feedback comments, marking responses against
outcomes and professional competency
frameworks
• Students receive feedback directly to their phones
without the need to be on campus or meet their
tutors face-to-face
©2013 MyKnowledgeMap Ltd 4

5. Inspired assessment and learning technology
What have we learnt from
this?
©2013 MyKnowledgeMap Ltd 5

6. Inspired assessment and learning technology
Boost student satisfaction
Improved tutor-student feedback loops can increase communication, engagement and
overall student satisfaction.
In 2011, the NSS (National
Quick, easy and immediate tools for providing tutor Student Survey) found that 97%
feedback can: of final year medical students at
• Maximise teaching within time-bound assignments the University of Leeds were
satisfied with their experience -
• Support students with moment-of-need feedback an increase of 12% from the
• Spread tutor workload to facilitate higher quality previous year. This was the first
cohort of final year students to
feedback - lots of regular comments rather than a
bulk at the end of a placement have been equipped with iPhones
and mobile assessments.
©2013 MyKnowledgeMap Ltd 6

7. Inspired assessment and learning technology
Enable “capturing the moment”
Allow students to capture valuable moments and experiences by providing assessments
that can be completed anytime, anywhere.
40
No. of assessments submitted
This approach has: Placement #1 Placement #2
32
• Spread student workload,
improving the quality of 24
submissions and responses 16
• Allowed students to make use 8
of all “assessable” situations in
the workplace 0
1 5 9 13 17 21 25 29 33 37 40
Placement Days
©2013 MyKnowledgeMap Ltd 7

8. Inspired assessment and learning technology
Within each placement students are
only required to submit 5 responses. No. students responding more than 5 times
2009 / 10 2010 / 11
But, when provided with tools to
Paediatrics 0 37 (15% of all students)
complete assessments anytime and
anyplace, students began to submit GOSH 0 23 (9.6% of all students)
more responses than were required. CCCC 37 (15.5% of all students) 56 (23% of all students)
This increased student engagement, Psych 0 37 (15% of all students)
quality of evidence, quality of feedback Primary care 0 23 (9.6% of all students)
and overall satisfaction.
©2013 MyKnowledgeMap Ltd 8

9. Inspired assessment and learning technology
Encourage good habits “
The iPhone changed everything. The medical
school uses it in two main ways. The first is as an
information resource. Students are given
Developing positive behaviours during education electronic copies of clinical handbooks and the
can serve students well downstream. British National Formulary. In the modern age,
when white coats and their accompanying deep
pockets are becoming a distant memory, this is
an important function in itself.
Easy to use and easy to access means:
However, it’s the second use—to support clinical
• Students are submitting work in a more timely learning—that I find most interesting. Gareth
fashion, managing their time more effectively tells me that a dedicated app allows students to
upload reflections on cases they have seen (with
• Students are making more notes, capturing more sensitive information removed) to an online
opportunities and becoming more reflective portfolio in real time. Workplace-based
assessments can also be completed on the spot
• Students see immediate benefit in being able to by clinicians: in short, the iPhone is helping
the medical school to provide consistent
receive prompt feedback - motivating them to support for students, and to get them into
engage more good habits for the future.
Niall Boyce writing in The Lancet
vol. 379, issue. 9812, page. 209 - 21 Jan 2012
©2013 MyKnowledgeMap Ltd 9

10. Inspired assessment and learning technology
Engage all stakeholders
Engaging a range of people in the assessment cycle can enrich student experiences
and validate captured evidence.
• Capture observations when and wherever they may Students are becoming more
engaged and the process is “upping
occur
the stakes” of clinical encounters,
• Provide validation of captured evidence - not just promoting improved performance.
self-assessed experiences
Students value seeing the positive
When done well you can even involve “non-registered” feedback loops and are becoming
individuals who are typically excluded from the e- more confident, adopting a self-
assessment process: select attitude - asking “can I do...?”
• Peers, managers, practice professionals, mentors, and taking advantage of assessable
situations.
coaches - even customers and service users
©2013 MyKnowledgeMap Ltd 10

11. Inspired assessment and learning technology
Manage quality & consistency of tutor input
Supporting teaching staff is key to software adoption and tools to help with prioritisation
of time is a great place to start.
“
During the early days of implementing e-
Usability and quality of design is key: assessment within the delivery of the
• Highly visible alerts and notifications can draw
Foundation Degree in Retail Management,
students began to find the system such a good
attention to priority areas way of getting feedback from their tutors that
they began to ask for help as many as 5 times an
• Clear activity summaries can alert when students evening.
need attention or extra support
With each tutor supporting around 120 students
• Easy to read dashboards can help tutors manage this soon overwhelmed them and even led the
universities (Manchester Metropolitan &
their own SLAs for student feedback University of the Arts, London) to create a
service level agreement for tutor feedback to
Gathering quality metrics and management support prioritisation of teaching effort.
information can also help to demonstrate quality of Tom Holland
Technical Marketing Manager, MyKnowledgeMap Ltd
teaching, quality of feedback and value for money.
©2013 MyKnowledgeMap Ltd 11

12. Inspired assessment and learning technology
Any questions?
dave.waller@myknowledgemap.com
http://www.myknowledgemap.com
@d13design @myknowledgemap
©2013 MyKnowledgeMap Ltd 12