This presentation takes a look at five tech solutions to qualitative research data collection and evaluates data quality associated with each solution based on the Credibility component of the Total Quality Framework.

1. Does the End Justify the Means?
Exploring & Evaluating
Technological Alternatives to
Qualitative Data Collection
Margaret R. Roller, MA
May 17, 2018

2. Does the End Justify the Means? – May 17, 2018 1Margaret R. Roller
 Qualitative research & typical methods/mode
 Tech solutions:Advantages & drawbacks
 Tech solutions: Method-mode alternatives
Overview

3. 2Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Reference: Total Quality Framework

4. 3Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Scope
Representation
Cooperation
Data Gathering
Validity
Researcher effects
Participant effects
Reference: TQF Credibility Component

5. 4Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Reference: Text
Roller & Lavrakas, 2015. New York: Guilford Press.
http://bit.ly/TQFbook

6. 5Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Reference: Blog
http://www.researchdesignreview.com

7. 6Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
What is Qualitative Research?Qualitative Research: Unique Attributes

8. 7Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Tech Solutions – Advantages
 Representation & cooperation
 Coverage & access (geography & population segment)
 Convenience & flexibility for participants
 Appropriate for sensitive topics
 Relevant form of communication, esp., for certain demographics
 Data accuracy & depth
 Mitigates memory/recall error
 Added dimension via blending approaches, use of visual
elicitation, & multimedia – text, video, images
 Efficient use of resources

9. 8Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Tech Solutions – Potential Drawbacks
 Internet access & comfort with technology
 Managing engagement (skills, time)
 Important visual &/or verbal cues may be missing
 Analysis – Lots of data (in volume & formats)
 Fraud, misrepresentation
“Identity is fluid and potentially multiple on the Internet.”
 Security, confidentiality, anonymity

10. 9Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Tech Solutions:
In-depth Interviews

11. 10
Tech Solutions: In-depth Interviews
EMAIL
WEBCAM
MOBILE
VIDEO
SensitiveTopic
Professionals
Diary/Life Story
Substitute For In-person
Usability
Concurrent Mixed Methods
In-the-moment Research
Location-specific, Geofencing
Study a Process or Event

12. 11
Email
SCOPE DATA GATHERING
CooperationRepresentation
Validity
Researcher
Effects
Participant
Effects
Comfort level
with mode
Length &
perceived
burden may lead
to drop outs
Exclusion of
some segments,
e.g., 65+ & <HS
Depth,
complex data
from follow-up
& “homework”
Participant’s
words & how
text is typed
may bias
Willingness to
discuss sensitive
topic due to
privacy, safe
environment
Mitigates social
desirability
responding
Mitigates
bias due to
ability to
reflect
Tech Solutions: In-depth Interviews
Access to pop
segment
Flexibility of
location
Geographic
coverage

13. 12
Webcam
Cooperation Validity Participant
Effects
Flexibility of
location
Requires
reminders &
instructions
Social desirability
Depth, ability
to showing
stimuli,
participant
markup
Researcher
Effects
Requires access
& technical
know-how
Willingness to
respond due to
rapport & cues
Consistency,
ability to
follow guide
Bias due to
visual cues
Observers &/or
recording(s) may
stifle responses
SCOPE DATA GATHERING
Representation
Tech Solutions: In-depth Interviews
Access to pop
segment
Geographic
coverage

14. 13
MobileVideo
Tech Solutions: In-depth Interviews
Cooperation Validity Participant
Effects
Engagement
with video
technology
Requires
reminders &
instructions
Selection bias,
choosing
what to share
Mitigates
recall error
Researcher
Effects
Access & comfort
with app –
downloading & use
Geographic
coverage Mitigates on-
site social
interaction
bias
Depth from
verbal, nonverbal,
& contextual cues
Inconsistency,
difficulty
following guide
Altered
behavior
Absence of
broader context
& meaning
SCOPE DATA GATHERING
Representation
Access to pop
segment

15. 14Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Tech Solutions:
Asynchronous Focus Groups

16. 15Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Tech Solutions: Asynchronous FGs
Asynchronous
Focus Groups
SensitiveTopic
Professionals
Hard-to-reach, disabled

17. 16
Cooperation Validity
Participant
Effects
Flexibility of
location, time,
& device
Requires
reminders &
instructions
Nonresponse –
Unwillingness or
inability to
participate in a
mostly text-
based discussion
Breadth & depth
due to days of
discussion
Researcher
Effects
Access &
comfort with
platform
technology
Platform
features help
mitigate bias
& maintain
consistency
Triangulation,
multi-media
responses – text,
video, image Losing focus
while
managing
engagement
Mitigates bias,
multiple ways
to respond
Representation
Tech Solutions: Asynchronous FGs
Access to pop
segment
Geographic
coverage
SCOPE DATA GATHERING

18. 17Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Platform Functionality – Discussion Guide
Tech Solutions: Asynchronous FGs

19. 18Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Platform Functionality – Discussion Guide
Tech Solutions: Asynchronous FGs

20. 19Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Platform Functionality – Discussion Guide
Tech Solutions: Asynchronous FGs

21. 20Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Platform Functionality – Activities
Tech Solutions: Asynchronous FGs

22. 21Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Platform Functionality – Managing Engagement
Tech Solutions: Asynchronous FGs

23. 22Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
Tech Solutions:
Observation &
Multimethod/mode

24. 23Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
“Day in the Life”
Journey
Specific assignment/task
Tech Solutions:
Observation & Multimethod/mode

25. 24
Mobility Patterns
Among Children
IDIs
Observation
via GPS &
SMS questions
“Day in the Life”
IDIs
using GPS maps &
guided tour as stimuli
AcneTreatment
2-month
multimedia diary
IDIs
via mobile video
+
Brazilian Migrants
6-month online
observation of
migrant communities
IDIs
via email & in-person
+
Tech Solutions:
Observation & Multimethod/mode

26. 25
Influence of the Internet
in the Classroom on Literacy
Email IDIs
Diaries
Photos
Journey
+
+
Teachers’ published articles
+
Emotional Effects of
Selecting a College
Webcam IDIs
Participants submit text, images,
video during the course of the
application-selection process
Asynchronous focus group
Tech Solutions:
Observation & Multimethod/mode

27. 26
Shopping For & Using Body
Wash Product
Audio responses to
questions on “wallet card”
In-store photos & videos
Specific Assignment/Task
+
In-shower audio responses
to questions on laminated card
In-the-moment Sensorial
Experience with Meatless Burgers
Photo + text
regarding package
1’ video + text
regarding interaction with package &
cooking burger (look, smell, texture)
Text
regarding the eating experience
Tech Solutions:
Observation & Multimethod/mode

28. 27
Tech Solutions:
Observation & Multimethod/mode
SCOPE DATA GATHERING
Cooperation Validity
Participant
Effects
Comfort with
technology
(among some)
Tasks/length
may be
burdensome Triangulation,
methodological
& researcher
Researcher
Effects
Bias towards
tech savvy
individuals
Broad geographic
coverage Mitigate bias
by reflecting
on multi data
Depth due to
synergy of
methods, multi-
layered data
Tasks bias
normality &
taint study
environment
Tasks/length
help reflexive
responding
Tasks alters
behavior &
attitudes
Selection bias
Representation

29. Summary – Key Strengths

30. Summary – Key Limitations

31. Does the End Justify the Means? – May 17, 2018
Interested in Qualitative Research
& Hard-to-reach Participants?
Recruiting Hard-to-Reach Respondents with Complex
Characteristics for Qualitative Interviews
Daniela Glusberg, Dánae Corado, Casey Langer Tesfaye
AAPOR Thursday May 17th
Time: 4:15 p.m. – 5:45 p.m. Session 10
Location: Governor’s Square 17
Check Out

32. Does the End Justify the Means? – May 17, 2018
Questions &
Comments
Margaret R. Roller, MA
Email: rmr@rollerresearch.com
Website: rollerresearch.com
Blog: researchdesignreview.com

33. Does the End Justify the Means? – May 17, 2018
Appendix
References

34. 33Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
EMAIL IDI
Bowker, N., & Tuffin, K. (2004). Using the online medium for discursive research about people with disabilities. Social Science
Computer Review, 22(2), 228–241.
Egan, J., Chenoweth, L., & McAuliffe, D. (2006). Email-facilitated qualitative interviews with traumatic brain injury survivors: A
new and accessible method. Brain Injury, 20(12), 1283–1294.
Hawkins, J. E. (2018). The Practical Utility Suitability of Email Interviews in Qualitative Research The Practical Utility Suitability of
Email Interviews in Qualitative Research, 23(2), 493–501.
McCoyd, J. L. M., & Kerson, T. S. (2006). Conducting intensive interviews using email: A serendipitous comparative opportunity.
Qualitative Social Work, 5(3), 389–406.
Stacey, K., & Vincent, J. (2011). Evaluation of an electronic interview with multimedia stimulus materials for gaining in depth
responses from professionals. Qualitative Research, 11(5), 605–624.
ASYNCHRONOUS FOCUS GROUP
Tates, K., Zwaanswijk, M., Otten, R., van Dulmen, S., Hoogerbrugge, P. M., Kamps, W. A., & Bensing, J. M. (2009). Online focus
groups as a tool to collect data in hard-to-include populations: Examples from paediatric oncology. BMC Medical Research
Methodology, 9(1), 15.
Harmsen, I. A., Mollema, L., Ruiter, R. A., Paulussen, T. G., de Melker, H. E., & Kok, G. (2013). Why parents refuse childhood
vaccination: a qualitative study using online focus groups. BMC Public Health, 13, 1183.
References

35. 34Margaret R. RollerDoes the End Justify the Means? – May 17, 2018
OBSERVATION & MULTIMETHOD/MODE
Christensen, P., Mikkelsen, M. R., Nielsen, T. A. S., & Harder, H. (2011). Children, mobility, and space: Using GPS and mobile
phone technologies in ethnographic research. Journal of Mixed Methods Research, 5(3), 227–246.
Hancock, K. (2012). Online qual studies health care foundation to shift its focus. Quirk’s Marketing Research Review, October,
Article ID 20121006.
Karchmer, R. A. (2001). The journey ahead: Thirteen teachers report how the Internet influences literacy and literacy instruction
in their K-12 classrooms. Reading Research Quarterly, 36(4), 442–466.
Schrooten, M. (2012). Moving ethnography online: Researching Brazilian migrants’ online togetherness. Ethnic and Racial
Studies, 35(10), 1794–1809.
OTHER
James, N., & Busher, H. (2009). Online interviewing. London: Sage Publications.
“Minute Maid Debuts New Campaign That Celebrates the Good in Everyday Family Moments,”
https://www.businesswire.com/news/home/20180409006050/en/Minute-Maid-Debuts-New-Campaign-Celebrates-Good
(Accessed April 10, 2018)
http://www.pewinternet.org/fact-sheet/mobile/ (Accessed April 17, 2018)
References