Getting started with your 2020/21 digital experience insights surveys
1. Getting started with the
Digital experience
insights (DEI) surveys
2020/21
September 2020
Helen Beetham, Tabetha Newman
Sarah Knight and Ruth Drysdale
2. 1. The DEI surveys: background (SK)
2. New for 2020 / post-Covid (HB)
3. Using DEI as a ‘pulse’ survey (TN)
Questions and feedback (voice, chat)
3. Detail: demographics, participation (TN)
4. Detail: online learning/teaching/working (HB)
Questions and feedback (voice, chat)
5. Next steps (RD)
Outline
3. • Our insights surveys provide powerful data on how students,
teaching and professional services staff (and now
researchers) are using technology
• Designed to help to understand and improve the digital
experience in FE and HE
• Provide baseline and benchmarking data to inform your
digital strategy
• Annual summative report on each survey for each sector,
providing vision and leadership
• New question sets for 2020-1 surveys running from October
and will reflect requirements for the COVID-19 context
• https://digitalinsights.jisc.ac.uk/what-is-digital-insights-
experience/
Digital experience insights surveys
4. The survey questions
Around 30 questions in four thematic areas:
• You and your technology
• Technology at your university/college
• Technology in your learning (or teaching,
research or work role)
• Developing your digital skills
Questions equivalent/mapped across users
Easily visualised in real time via user dashboard
Response data shown to be relevant, actionable and
comparable
https://digitalinsights.jisc.ac.uk/what-is-digital-insights-experience/
6. Pre-Covid plans for 2020/21
• Develop and pilot a new question set for researchers
• Analyse existing and new questions
• Refine and update all question sets
• Map questions more closely across all four question
sets
• Maximise continuity for loyal users
7. Emergent findings on impact of COVID-19
Pivot to new practice – rapid learning (opportunities and
stresses), training and workload issues
Issues of mental health and wellbeing online
Motivation and engagement
New and widening inequalities e.g. hardware, networks,
data costs, home space
Quality of and access to remote systems
Quality of and access to resources/materials
Safety and privacy issues
8. Post-Covid plans for 2020/21
• ‘Experience’ happens in a shorter time frame
• Mode of participation is key – but can change
• Iterative data collection (tracking change within the year)
• Regular reporting and sharing rather than summative
• Adapt Qs to focus on online and post-Covid issues
• Space for feelings/wellbeing issues and negative reactions
• Many surveys, panels, focus groups etc – how to add value?
• Build on everything we have learned - but continuity less critical
than supporting organisations through crisis and change
9. Post-Covid plans for 2020/21
• ‘Experience’ happens in a shorter time frame
• Mode of participation is key – but can change
• Iterative data collection (tracking change within the year)
• Regular reporting and sharing rather than summative
Adapting the service
Adapting the questions
• Adapt Qs to focus on online and post-Covid issues
• Space for feelings/wellbeing issues and negative reactions
• Many surveys, focus groups etc – how can DEI add value?
• Build on everything we have learned - but continuity now less
critical than supporting organisations through crisis and change
11. Participation, context, expectation, time frames
How people participate (online/onsite/blended) is crucial to
their whole experience
• We need to be able to partition or filter data using this factor
Context likely to change
• Personal circumstance (e.g. in quarantine, shielding)
• Shifting policies and priorities: course, organisation, national
• Online mode may be personal or group (cohort, team)
Expectation likely to drive attitudes
• Choosing online vs enforced/emergency online
Some questions are time-bound (e.g. ‘weekly’, ‘monthly’)
• Chose ‘last two weeks’ for all questions
13. What is a ‘pulse survey’
• One survey, run for short time periods, repeated soon after to
identify and resolve issues
• Very useful now that the learning, teaching and working experience
has changed and is changing unpredictably
• Questions refer to shorter time frame: “in the last two weeks”
• Can be used with the same set of people, or with different cohorts
• Can be run in discrete windows (e.g. start of year, just before
Christmas, just before Easter), or continuously
14. Using DEI surveys as ‘pulse surveys’
Oct.2020
30April
2021
Term 1 Term 3Term 2
2
weeks
2
weeks
2
weeks
College
A
Uni’ B 2
weeks
2
weeks
2
weeks
2
weeks
2
weeks
2
weeks
16. Follow the ‘six steps’ as before…
1. Log in
2. Locate your surveys on your dashboard
3. Design/customise (add a logo and any questions of your own)
4. All surveys open (October 2020)
5. Check and launch
6. Promote and engage
17. But also…
1. Log in
2. Locate your surveys on your dashboard
3. Design/customise (add a logo and any questions of your own)
4. All surveys open (September 2020)
5. Check and launch
6. Promote and engage
7. Visualise your data at any time using the live ‘analyse’ window
8. Filter by date at any time
9. Promote and engage again to collect further data
10.Close, download and filter all data (all surveys close by 30 April 2021)
19. Issues to consider if you use DEI as ‘pulse surveys’
• Useful to see responses over time and track changes
• If surveying the same people at different times, analyse only within each
time frame – it is not valid to group data across the year
• If surveying different people in each date range, make sure your sample
is big enough to minimize random effects
• We expect so much change over the year that a summative view (adding
all the data across the year) is unlikely to be meaningful
• This means annual benchmarking may be of limited value (to discuss…)
20. Full ‘getting started’ guide: coming soon
https://digitalinsights.jisc.ac.uk/running
-insights-surveys/getting-started/
22. Adapting the DEI questions: focus on
learning/teaching/working online
23. Theme 1: You and your technology
Focus on those issues we know (from DEI and other surveys)
create and widen digital inequalities
24. Theme 2: Technology in your organisation
Physical infrastructure on site matters less than online
Qual analysis 2019/20 found ‘learning environment’ was a
complex and evolving idea
A lot of confusion about separate platforms and functions
But respondents do know what qualities they want
29. Student digital experience insights survey 2020: UK
higher education (HE) survey findings:
https://www.jisc.ac.uk/reports/student-digital-
experience-insights-survey-2020-uk-higher-education
Learner digital experience insights survey 2020: UK
further education (FE) findings:
https://www.jisc.ac.uk/reports/learner-digital-
experience-insights-survey-2020-uk-further-education
Student digital experience insights survey 2020:
question by question analysis of findings from
students in UK further and higher education:
https://www.jisc.ac.uk/sites/default/files/dei-2020-
student-survey-question-by-question-analysis.pdf
30. Sign up to run the surveys https://digitalinsights.jisc.ac.uk/subscribe/find-out-more/
See our guidance https://digitalinsights.jisc.ac.uk/running-insights-surveys/
Join our Insights community: Click on jiscmail.ac.uk/JISC-DIGITALINSIGHTS-
COP and join by clicking on the 'Subscribe or Unsubscribe' button
Follow #digitalstudent
Date for your diary – 17th November for our online community of practice event
Get involved
31. Except where otherwise noted,
this work is licensed under CC-BY
Get in touch…
Digital experience insights
Sarah Knight
Sarah.knight@jisc.ac.uk
Ruth Drysdale
Ruth.drysdale@jisc.ac.uk
https://digitalinsights.jisc.ac.uk
#digitalstudent
Notas del editor
I have slightly rewritten these slides, Sarah
We will share with you the key findings from each of the four themes and highlight some implications for consideration for the new academic year.
(remove redundant questions, include powerful/actionable new questions)
Plethora of recent surveys in this space
Unions
National interest
Emergency online is different to elective online
Online experience – was digital/blended
Note that mode = online, on-site, many blends in between.
Summative: refer to benchmarking and reporting. These are issues we will come back to.
Everything we have learned: (digital focus, question design, actionable results, 360 viewpoint)
Online experience – was digital/blended
Note that mode = online, on-site, many blends in between.
Summative: refer to benchmarking and reporting. These are issues we will come back to.
Note that these decisions have knock-on consequences that we will return to.
Helen
Note that these decisions have knock-on consequences that we will return to.
Helen
This was wordy compared with other slides – tightened up
Survey stays open
Can be used to collect data once – like previous years – or as a pulse survey - a tool to collect quick snapshots of information
Visualise your data at any time using the live ‘analyse’ window
Close data collection and benchmark/download at any time
Relaunch to collect further data at a later time
Visualise your data at any time using the live ‘analyse’ window
Close data collection and benchmark/download at any time
Relaunch to collect further data at a later time
This was a new question added to the survey for 2019-20. Although 76 % of students were either very or quite confident at trying out new technologies, only 43 % were comfortable with using mainstream technologies.
Students who participated in Jisc consultation events in July echoed the messages that we cant assumer that all students feel comfortable or confident in using the technologies provided.
One-stop-shop apps/portals mean users struggle to differentiate systems (e.g. LMS/library)
Users mix organisational w personal apps and interfaces (sometimes encouraged to do so)
This year, there are many new platforms in use
Users have to mix and match e.g. to compensate for poor bandwidth, changes of location
Most users answer as if this means:
Organisational platforms (vs personal apps)
Online learning resources – provided by e.g. the library
Organisational networks (vs personal wifi/mobile networks)
But there
Note that all the examples are from the student build
This was one of our key design tasks this year – to ensure we included a wide range of learning activities. And again, learning ‘resources’ or ‘materials’ is a very broadly understood term, but learners in particular know when they are getting something useful, and when they are not.
Again note that all the examples are from the student build
We wanted to avoid any questions that asked about the on-site experience, as we feel that is being so widely covered including in the national press, and we wanted to focus on what we do best. So we do ask if students feel safe learning online, and we ask staff if they feel safe teaching, working or researching online. We give them free rein to talk about the positive and negative – important this year when the situation is so new. Next year we may have enough information to use closed questions here, as with the learning materials.
Note that there are other changes but as much continuity as possible.