1.
Getting most out of your SENSORS
Mixed-Methods Research Methodology Enabling Identification, Modeling
and Predicting Human Aspects of Mobile Sensing "In the Wild"
Quality of Life Technologies Lab
University of Geneva & University of Copenhagen
qualityoflifetechnologies.org
Alexandre De Masi, Prof. Katarzyna Wac

2.
Agenda
▪ Who are we ?
▪ “In the Wild”
▪ Mixed-Methods
▪ Use Cases
▪ Wrangling, Modeling and Predicting
▪ Conclusive Remarks

3.
World Health Organization | www.who.int
Who are we ?

4.
Patients
mQoL
Plateforme flexible pour
études sur la santé
humaine
Researcher
Clinicians
Instituts
Visualization
Exploration
Modelization
Smartphones
Wearables
Calibrated scales (PROs)
EMR
• • • • • •
• • •
Passive Data Collection
Patients’ Data
Mobile Data Platform
What is our goal ?

5.
Bending the Curve
Wac, K., Fiordelli, M., Gustarini, M., & Rivas, H. (2015). Quality of life technologies: Experiences from the field and key challenges.
IEEE Internet Computing, 19(4), 28-35.
Fries, J. F. (2002). Aging, natural death, and the compression of morbidity. Bulletin of the World Health Organization, 80, 245-250.
QoL Domains QoL Facets (13/24)
Physical
health
Activities of daily living ✓
Dependence on medicinal substances and medical aids
Energy and fatigue ✓
Mobility ✓
Pain and discomfort ✓
Sleep and rest ✓
Work capacity
Psychological
Bodily image and appearance
Negative feelings ✓
Positive feelings ✓
Self-esteem
Spirituality / Religion / Personal beliefs ✓
Thinking, learning, memory, and concentration
Social
relationships
Personal relationships ✓
Social support ✓
Sexual activity
Environment
Financial resources
Freedom, physical safety, and security ✓
Health and social care: accessibility, and quality
Home environment
Opportunities for acquiring new information and skills
Participation in and opportunities for recreation / leisure activities
Physical environment (pollution / noise / traffic / climate) ✓
Transport ✓
{
{
{
{
Computational
Models
Self-management &
Behaviour Change
Facilitation
1000+ Participants
(mQoL Living Lab)
World Health Organization | www.who.int
“The World Health Organization Quality of Life Assessment (WHOQOL): development and general
psychometric properties.,” Soc. Sci. Med., vol. 46, no. 12, pp. 1569–85, Jun. 1998.

6.
“In the Wild”
In-situ design, development and evaluation
- Real-time
- Minimally intrusive
- Context/Environment is everything
- During multiple months or years
Chamberlain, A., & Crabtree, A. (Eds.). (2020). Into the Wild: Beyond the Design Research Lab. Springer International Publishing.

7.
Mixed-Methods : Quantitative & Qualitative

8.
Mixed-Methods : Quantitative & Qualitative
“What humain aspect are we exploring ?”
“ What does the literature says about this aspect ?”
- Quantitative : Objective, continuous, automatic,
unobtrusive, sensory based, in-context
- Qualitative: Subjective, momentary, self reported,
memory based, biased

9.
Explorative, Inductive &
Hypothetico-Inductive Approach
‘digital phenotyping’

10.
Dey, A. K., Wac, K., Ferreira, D., Tassini, K., Hong, J. H., & Ramos, J. (2011). Getting closer: An Empirical Investigation Of The Proximity Of User To Their Smart Phones. In Proceedings of the ACM UBICOMP.
Smartphone
as a sensor
88%
of the time
next to us
1 /4
Quantitative

11.
De Masi, A., & Wac, K. (2018). You're Using This App for What?: A mQoL Living Lab Study. Mobile Human Contributions: Workshop in conjunction with ACM UBICOMP, Singapore, October 2018.
Berrocal, A., Manea, V., De Masi, A., Wac, K. Step-by-Step Creation of a Flexible Platform to Conduct Studies Using Interactive, Mobile, Wearable and Ubiquitous Devices, ACM MobileHCI 2020 (under evaluation)
Smartphone
Logger
mQoL-log
6.6+ billion
data points
Quantitative

12.
Wac, K. (2018). From Quantified Self to Quality of Life, Chapter in: Digital Health: Scaling Healthcare to the World, Series: Health Informatics, Springer Nature, Dordrecht, the Netherlands.
Annotated wearables dataset https://doi.org/10.6084/m9.figshare.9702122
Wearables
438 wearables (2018)
Quantitative

13.
What About Qualitative Methodology ?
- Initial and Final interview
- ESM/EMA
- DRM

14.
Hyewon Suh, Nina Shahriaree, Eric B. Hekler, and Julie A. Kientz. 2016. Developing and Validating the User Burden Scale: A Tool for Assessing User Burden in Computing Systems. In Proceedings of the 2016 CHI Conference on Human Factors in
Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 3988–3999. DOI:https://doi.org/10.1145/2858036.2858448
2 /4
Qualitative: Initial & Final Interview
Initial : Before the Study
- Since when use of mobile, what phone(s) & provider
- Specific questions depending of the human aspect researched
- Phone and wearable proximity at day-night, week-weekend, home-outside, etc
- Age, education, occupation, marital status, cultural background, etc
- Any other comments
Final : After the Study
- Experience of the participants : User Burden Scale
- Comments about the smartphone and wearable (HCI)
- Anecdotal stories (out-of-the-scope)

15.
Intille, Stephen et al. “μEMA: Microinteraction-based Ecological Momentary Assessment (EMA) Using a Smartwatch.” Proceedings of the ... ACM International Conference on Ubiquitous Computing . UbiComp (Conference) vol. 2016 (2016): 1124-1128.
doi:10.1145/2971648.2971717
3 /4
Qualitative : ESM/EMA
ESM : Experience Sampling Method
EMA : Ecological Momentary Assessment
Registering experience at a given moment
In-situ survey triggered by:
- Pre-established/Random times/intervals
- “Everyday around 8:00 am”
- Context based: “Once the study subject arrived at work”
- Enabled by : sensor data fusion (e.g., WiFi, heart
rate, …)
Subjective: like “an-ad-hoc diary” (no need to memorize), may
miss context

16.
Kahneman D, Krueger AB, Schkade DA, Schwarz N, Stone AA. A survey method for characterizing daily life experience: the day reconstruction method. Science. 2004 Dec 3;306(5702):1776-80. doi: 10.1126/science.1103572. PMID: 15576620.
4 /4
Qualitative : DRM
DRM : Day Reconstruction Method
- Specific episodes of last 24h “re-instantiated” in memory
in some detail.
- Indicate approximate times (and/or estimate duration)
- Where, Who with, What - additional questions tailored
for the study.
Characteristics:
- Subjective, like “cumulative diary”
- May miss context if additional questions not asked
- Depends on organization of memories, memory bias /
memorable episodes
- May depend on social desirability of answers

17.
Adding Context to Subject Reports
InFrequent
Qualntitative
SuObjective
Memory
Sensory
Biased
Socially
Non-
Judgemental
Desirable
!
!
!
!!
!
!
!
!
!
!!
!
!! Context-Rich
Continuous
Longitudinal
! !! !! !
!

18.
mQoL Lab: Step-by-Step Creation of a Flexible Platform to Conduct Studies Using Interactive, Mobile, Wearable and Ubiquitous Devices A Berrocal, V Manea, A De Masi, K Wac - Procedia Computer Science, 2020
mQoL-Lab In Reality: Tech View

19.
Concrete Cases
- Sleep
- Stress

20.
Use Case :
Sleep
“During the past month,
How often have you had
trouble sleeping because you”
Wake up in the middle
of the night or early
In the morning?
Buysse, D. J., Reynolds III, C. F., Monk, T. H., Berman, S. R., & Kupfer, D. J. (1989). The Pittsburgh Sleep Quality Index: A new instrument for psychiatric practice and research. Psychiatry Research, 28(2), 193-213.

21.
From a Wearable...
Wac, K., Montanini, L., Ryager, K. B., & Rivas, H. (2017). Digital Health Tools for Sleep Self-Management: Working Mothers Use Case. In 38th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine (SBM 2017), USA, 2017.
User
ID
Hours of sleep User
ID
Hours of sleep
Weekday Weekend Weekday Weekend
2 5.5 +/- 13 min. 6 +/- 20 min. 5 6 +/- 18 min. 8.25 +/- 15 min.
3 7 +/- 17 min. 7.3 +/- 23 min. 7 7 +/- 15 min. 8 +/- 15 min.
4 8 +/- 10 min. 6.25 +/- 30 min. 10 6 +/- 10 min. 5.75 +/- 10 min.
N = 6 working mothers (DK)
Up to 6 months each
Study details

22.
Spring Summer Fall Winter
Laghouila, S., Manea, V., Estrada, V., Wac, K. (2018). Digital Health Tools for Chronic Illness and Dementia Risk Assessment in Older Adults, 39th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine (SBM 2018), USA, 2018.
N = 75 seniors over 65 (HU, ES)
Enrolled since January 2017 for at least 6 months
Study details
Currently: multivariable logistic regression models for PSQI (PRO) vs. wearable dataset (TechRO)
Via a Wearable & Risk Assessment

23.
...Back to Phone (ON-OFF ‘Sensor’)
N = 14: working mothers (S2-S5) and students (S11-S20), up to 6 months each (DK)
Ciman, M., Wac, K. (2019), Smartphones as Sleep Duration Sensors: Validation of the iSenseSleep Algorithm, JMIR mhealth and uhealth, Vol 7, No 5.
User ID Sleep duration Phone ON-OFF Remark
BASIS watch
[avg±std]
Estimate [avg±std]
* significantly diff
2 (mothers) 393 +/- 10 min. 418 +/- 14 min.
3 402 +/- 15 min. 507 +/- 16 min.* Over
4 455 +/- 13 min. 377 +/- 34 min.* Under
5 446 +/- 15 min. 444 +/- 25 min.
11 (students) 429 +/- 15 min. 481 +/- 23 min.* Over
12 473 +/- 16 min. 478 +/- 25 min.
13 377 +/- 22 min. 377 +/- 24 min.
14 450 +/- 15 min. 454 +/- 35 min.
15 482 +/- 14 min. 459 +/- 24 min.
16 478 +/- 16 min. 446 +/- 36 min.
17 409 +/- 19 min. 378 +/- 45 min.* Under
18 417 +/- 24 min. 374 +/- 16 min.
19 462 +/- 16 min. 346 +/- 25 min.* Under
20 452 +/- 29 min. 426 +/- 39 min.* Under
Study details
mQoL Log

24.
Stress
Ciman, M., Wac, K., (2018). Individuals’ Stress Assessment Through Human-Smartphone Interaction Analysis, IEEE Transactions on Affective Computing (IEEE TAC), 9(1): 51-65.
N = 38 participants, 1 month (CH)
Study details

25.
You have the method,
what about the data ?

26.
Wrangling, Modeling and Predicting
Best practices:
- Constant monitoring
- Testing the data as they come
- Backup rule of three
- Prepare for worst case scenario
Tools : Jupyter Notebook and R for data tasks
Machine learning:
- KISS: Keep It Simple St***d
- Oversampling, aggregation.
- Time Series analysis : forecasting, classification or regression.

27.
Conclusive Remarks
- Limitations:
- Limited time (3 years max)
- Limited number of participants
(aggregated 317 subject)
- Future work:
- Methodology paper
- Longitudinal study ( 2 to n-years) with Mixed-Methods to enable
predictive health.

28.
Exercice
Now you learned the Mixed-Method, let try to
prepare a study together using it.

29.
EXERCISE
- Define a use case: choose a human aspect to study
- Define the data to collect with:
- Smartphone
- Wearable
- Define the content of:
- Interviews
- ESM/EMA
- DRM
-