An exploration of the impact on the research, researcher, researcher and reader if a quantitative study were to have been done using a mixed methods approach instead.
A presentation created for EDCI 515
4. The Researcher:
Alicia O’Cathain
● Professor of Health Services Research and
Director of Medical Care Research Unit,
University of Sheffield
● Well-educated, older white woman
● Researches evaluation of new health services,
patient views of health care, urgent care
● Writes, teaches and uses mixed methods
4
6. Quantitative
Closed-ended info that results in a
numerical representation
.
Qualitative
Subjective and open-ended info that
allows the participant voice to be heard
6
Mixed Methods
+
8. Examples
Data Interview
Data is collected (i.e. using a
survey); participants are
interviewed to learn more about
some/all responses.
Interview Survey
Interview participants to learn
perspectives on or feelings about
a topic; use that info to develop a
survey.
8
Interview Survey
Conduct interviews to determine
whether participants are
appropriate for a quantitative
study; often used in health
research/RCTs.
Focus Group Survey
Host one or more focus groups
about a topic; validate responses
with a larger group via a survey.
9. ● Fills in the gaps when it comes
to weaknesses
○ Quantitative: Limited in
understanding of context
○ Qualitative: Can include
bias and difficult to
generalize
● Provides a more
comprehensive answer to the
research question
● Results can be validated within
the study
Advantages
● May require more
resources (time, people,
money)
● Research procedure is
more complicated (to
create and carry out)
9
Challengesvs.
12. Accessing online learning material:
.
Quantitative behavior patterns and their effects
on motivation and learning performance
12
Authors: Liang-Yi Li and Chin-Chung Tsai
Journal: Computers & Education
Date: July 2017
13. Quick Background
● Learning management systems (LMS) support
teachers in conducting a variety of activities.
● Over the years, researchers have analyzed LMS
system logs answer a number of questions and
improve online learning.
● Learning on an LMS is a self-regulated process.
13
14. Research Questions
14
What online learning materials did students spend more time on?
• Lecture slides, video lectures, shared assignments or posted messages
1
What kinds of behavioural patterns exist when students viewed
online learning materials?
2
How did the different behavioural patterns relate to students’
learning performance?
3
How did the different behaviour patterns relate to students’
learning motivation?
4
15. Method
Participants
59 third-year computer science
undergraduate students enrolled
in a blended course called Mobile
Phone Programming.
LMS usage
All clicking events were recorded
in the LMS system log; what they
are clicking on, when, from where,
for how long.
15
Learning performance
Evaluated via multiple homework
assignments each week
(completing a program) and a
final examination (paper based
and programming based)
Motivation
Measured by the Motivated
Strategies for Learning
Questionnaire, which students
filled out via LMS.
16. Results
16
What online learning materials did students spend more time on?
• Lecture slides, video lectures, shared assignments or posted messages
1
What kinds of behavioural patterns exist when students viewed
online learning materials?
2
“Students accessed the lecture slides more often and, on average,
spent more time on each access than they did for other learning
materials.”
Researchers classified the students with similar viewing
behaviours into three groups: “consistent use” students, “slide
intensive use students” and “less use students”.
17. Results
17
How did the different behavioural patterns relate to students’
learning performance?
3
How did the different behaviour patterns relate to students’
learning motivation?
4
a) “Consistent use” and “slide intensive use” students had higher
homework and examination scores than “less use” students.
b) “Consistent use” students had higher homework scores than
“slide intensive use” students, but the same exam scores.
The “consistent use” students were higher in intrinsic goal
orientation, self-efficacy and task value.
18. 4.
My Paper as
Mixed Methods
What would happen if they did mixed
methods instead of quantitative?
19. Quantitative Mixed Methods
The Research
19
The Reader
The Researcher
The Researched
20. The Research
20
Learn the WHY behind online learning material choices1
Learn the why behind behavioural patterns2
a) Why did they make those choices re: online learning materials?
b) Why did students not highlight or comment on the online
materials?
Why did “less use” students use the content less? Are there
outside influences? Are there ways we could influence/entice use?
21. The Research
21
Validate the motivation data3
Explore how students are actually using the material4
Does motivation actually influence behavioural patterns, or are
there also other conflicting factors?
“It cannot be determined if a student is actively and effectively
learning from a material once they have accessed it.”
22. The Researcher
Currently
Liang-Yi Li: Professor,
Department of Computer
Science and Engineering,
National Central University,
Taiwan
Chin-Chuang Tsai: Professor
and Chair, Program of
Learning Sciences, National
Taiwan Normal University,
Taiwan
Future
More time and resources
Would likely require the
project to take on additional
collaborators:
○ To provide expertise on
the method
○ To provide help in
distributing the labour
22
23. The Researched
Who is being researched
would not change.
Still studying 59 third-year
computer science
undergraduate students
enrolled in Mobile Phone
Programming.
.
The impact on the
research participants
would change.
A subset of students would
have to spend time actively
engaging in the research
project.
23
24. The Reader
● Have a better understanding of the value of the
research and be able to apply it.
● What materials should I use/create?
● How should my course be set-up?
● What digital literacy skills might my
students be missing?
● How can I motivate my students?
24
26. Questions & Insights
● Mixed methods can really help you to
understand why something is happening.
● Eliminates the need for (some) future research
● How do you know when to stop going deeper?
● Can you switch to mixed methods part way
through a study?
26
27. References
27
Li, L.-Y., & Tsai, C.-C. (2017). Accessing online learning material:
Quantitative behavior patterns and their effects on motivation
and learning performance. Computers & Education, 114, 286–
297.
O’Cathain, A. (2010). Assessing the Quality of Mixed Methods
Research: Toward a Comprehensive Framework. In A.
Tashakkori & C. Teddlie, SAGE Handbook of Mixed Methods in
Social & Behavioral Research (pp. 531–556).
Overview of Mixed Methods. (n.d.). Retrieved July 10, 2019,
from Centre for Innovation in Research and Teaching website:
Editor's Notes
Professor of Health Services Research and Director of Medical Care Research Unit, University of Sheffield
Medical Care Research Unit focuses on the development and evaluation of complex interventions for chronic conditions (CF, MS, etc.)
Researcher, teacher and administrator
Based in England, so likely has a Western perspective
Well-educated, older white woman (as far as I can tell)
BSc, MSc, MA, PhD
Published her first paper in 1988
This could inform her perspective (is she clinging to a past? Moving forward with the times)
Researches evaluation of new health services, patient views of health care, urgent care
Does LOTS of research (so has lots of experience!). 11 current research projects listed on website, 13 publications listed in the last 5 years.
Much of her research uses mixed methods, and/or randomized control trials.
Writes, teaches and uses mixed methods
Helps explain why she is considered to be an expert on mixed methods for this textbook chapter.
4 different textbooks chapters, numerous papers focused on the use of applicability of mixed methods for certain situations
Mixed methods in in biomedical and health research
The role of group dynamics in mixed methods health research teams
Three techniques for integrating data in mixed method studies
Runs mixed methods workshops and clinics for PhD candidates
Based in England (western perspective)
Teacher, researcher, administrator
Unit focuses on the development and evaluation of complex interventions for chronic conditions.
11 current research projects, 13 publications in the last 5 years.
Puts the emphasis on the research question, and how to really get the answer
Uses all approaches available to come to a better understanding
Often allows us to better understand both the what and the why
It’s not really a simple math equation- both the quantitative and qualitative parts can influence and affect one another.
Different types of mixed methods design, which differ on timing, order of pieces, and how they influence each other
Example in mixed methods paper could happen concurrently, because they were pretty separate, but allowed for a much better answer to the question. (and findings that weren’t just misleading!!)
Data Interview: could be a survey, could be data pulled from an LMS, could be data pulled from social media
Interview survey: “why didn’t you participate” question on orientation survey
Interview survey (health): many randomized control trials, to ensure you meet criteria
Focus group survey: market research; could have done this with the women’s scholar online harassment study
Variety of activities
Online or blended course (or just extra)
Reading materials, asynchronous discussion (and synchronous discussion), quizzes, self/peer assessment
understand engagement levels
discover behaviour patterns
explore the relationship between behaviour and learning performance
Self-regulated process. It’s up to you to decide what course material to access, when, for how long; you control your learning
Consistent use = accessed more often for shorter periods, all types od materials
Slide intensive use = accessed less often but for longer; spent same time overall, but mostly on the slides.
Less use = just didn’t use as often
Obviously, need to add a qualitative component in order to achieve the mixed methods methodology.
According to the conclusion, “understanding how students view different learning materials and how their viewing behaviours affect their learning has been an important issue. “
But this study doesn’t actually determine student VIEWS in any way… just use. They may not be related .
We’re assuming views based on behaviour
A) why did they make these choices?
Did they spend more time on content they found more useful?
Did some content just TAKE less time (i.e. posted messages. Shared assignments if you’ve solved the assignment)?
Was the system set-up to encourage use of one over the other?
LMS provided a notification page where students could view how many posts they had not viewed- students then didn’t have to access??
Method of access
B) Why did they not highlight? YOU COULD LITERALLY JUST ASK THEM.
Not used to doing it
Didn’t like digital annotation tools
Copied and executed codes in the IDE; made comments there.
Also: didn’t know the tool existed??
2.
The behavioural patterns aren’t all based on choice… they are also based on total time spent. Why did “less use” students not engage with the content?
Is it all based on motivation? Did the know the system existed? Did outside factors (work, mental health) prevent them from engaging as much.
Are their outside influences on behavioural patterns (what’s happening outside the LMS? Backchannels? How are students perceiving their learning? Do they just think they already know it all?)
Are there any strategies that could have made a difference?
Liang Yi: probably only familiar with quantitative research, coming from computer science (and based on quick skim of other published articles)
Chin Chuang: educated in both science and education; likely not overly familiar with qualitative either (views, self-efficacy)
Require expertise, but also more resources
Interviews and analysing them takes a lot of time!!
Who wouldn’t change. They’re not recruiting participants, they have a set group.
Partly wouldn’t change because the qualitative part is being added AFTER the quantitative.