HCI and Smartphone Data at Scale

©2009CarnegieMellonUniversity:1
HCI and
Smartphone Data at Scale
IBM Research
July 29, 2013
Shah Amini
Justin Cranshaw
Afsaneh Doryab
Jialiu Lin
Jun-Ki Min
Jason Wiese
Jason Hong
Norman Sadeh
Joy Zhang
John Zimmerman
Computer
Human
Interaction:
Mobility
Privacy
Security

Smartphones are Pervasive
• 50% penetration in the
US mid-2012
• In 2013Q1, majority of
phones sold worldwide
• 50 billion apps
downloaded on each
of Apple and Android

Smartphones are Intimate
Mobile phones and millennials (Pew 2012):
• 75% use in bed before going to sleep
• 83% sleep with their mobile phones
• 90% check first thing in the morning
• Half use them while eating
• A third use them in the bathroom (!)
• A fifth check them every ten minutes

Smartphone Data is Intimate
Who we know
(contact list)
Who we call
(call log)
Who we text
(sms log)

Smartphone Data is Intimate
Where we go
(gps, foursquare)
Photos
(some geotagged)
Sensors
(accel, sound, light)

The Opportunity
• We are creating
a worldwide
sensor network
with these
smartphones
• We can now
capture and
analyze human
behavior at
unprecedented
fidelity and scale

Three Threads of Research
• Augmented Social Graph
– Create richer computational models
of our social relationships with others
• Urban Analytics
– Create viz and models of cities
based on geotagged social media
• CrowdScanning Apps
– Crowdsourcing and other techniques
to analyze privacy behaviors of apps

Modeling Social Relationships
• If you were in a jail in Mexico, which
of the 500+ “friends” in your phone
contact list and on Facebook would
come and get you out?

Modeling Social Relationships
• Can we use smartphone data to build
a richer augmented social graph?
– models tie strength, group, role

Why Better Models?
• Secure invitations
– Who is this person friending me?
• Communication triage
• Better info finding (weak ties)
• Configuration of privacy policies
– Tie strength strongly correlated with what
personal info people willing to share
(Wiese et al, Ubicomp 2011)
• Early detection of depression
– Less communication with strong ties,
less mobility, lots of fast food, insomnia

Ongoing Work: Sleep Data
Sleep data
(self-reported
ground truth)
Sensor data

Using Call Log, SMS, Contacts

User Study on Relationships
• 40 Participants
– 13 male and 27 female (age 19-50)
– 55% student, 35% employed, 10% unemp
• Data collection
– Phone: Contact list, call & SMS logs
– Facebook: Friend list from Facebook
backup
– Self-report for 70 contacts: Demographics,
group, closeness (1 – 5 = feel very close)

Life Facets
• Can classify life facets
{work, social, home}
at 90.1%
– If at least one comm.
– Just contact list,
call log, SMS log
• Correlations
Min et al, Mining Smartphone Data to
Classify Life-Facets of Social
Relationships, CSCW 2013

Ongoing Work: Tie Strength
• However, tie strength much harder to
predict, 74.6% for {low, med, high}
– We thought this would be easy…
– Other modes of communication
• Skype, IM, email, face-to-face
– Stage of relationship / maintenance comm.

• Urban Analytics

The Problem
• Today’s methods for gathering data about
cities are slow, expensive, and limited
– Ex. Travel Behavioral Inventory for traffic
flows every 10-20 years and 100s of people
– US Census 2010 cost $13 billion
– Quality of life surveys (sociology, city govts)
go door-to-door and interview people
• Some approaches today:
– Call Data Records, but granularity
– Deploy a custom app, but scale and utility

The Vision: Urban Analytics
• Goal: Use smartphones + social media +
machine learning to offer new and useful
insights about a city in a manner that is
cheap, fast, and highly scalable

Livehoods, Our First Urban
Analytics Tool
• The character of an urban area is defined
not just by the types of places found
there, but also by the people that make
it part of their daily life
Cranshaw et al, The Livehoods Project: Utilizing Social Media to
Understand the Dynamics of a City, ICWSM 2012.

What comes to mind when you
picture your neighborhood?

You’re probably not imagining this.
The Image of a Neighborhood

The Image of a Neighborhood
What you’re imagining probably looks a lot more like this.
Every citizen has had long associations with some
part of his city, and his image is soaked in
memories and meanings.
---Kevin Lynch, The Image of a City

Kevin Lynch, 1960 Stanley Milgram, 1977
Studying Perceptions:
Cognitive Maps

Two Perspectives
“Politically constructed” “Socially constructed”
Neighborhoods have fixed
borders defined by the city
government.
Neighborhoods are
organic, cultural artifacts.
Borders are
blurry, imprecise, and may be
different to different people.

Two Perspectives
“Socially constructed”
Neighborhoods are
organic, cultural artifacts.
Borders are
blurry, imprecise, and may be
different to different people.
Can we discover
automated ways of
identifying the “organic”
boundaries of the city?
Can we extract local
cultural knowledge from
social media?
Can we build a collective
cognitive map from data?

Livehoods Data Source
• Crawled 18m check-ins
from foursquare
– Claims 20m users
– People who linked their
foursquare accts to Twitter
• Spectral clustering based
on geographic and social
proximity

☺
☺
☺
☺
If you watch check-ins over
time, you’ll notice that groups
of like-minded people tend to
stay in the same areas.

We can aggregate these
patterns to compute
relationships between check-
in venues.

These relationships can then
be used to identify natural
borders in the urban
landscape.

Livehood 2
Livehood 1
We call the discovered
clusters “Livehoods”
reflecting their dynamic
character.

Try it out at livehoods.org

Evaluation
• Interviewed 27 locals
– Residents, urban planners, businesses
– Asked them to draw their mental maps of
areas first
– Then showed them our maps and solicited
feedback

South Side Pittsburgh

Carson Street runs along the length of
South Side, and is densely packed with
bars, restaurants, tattoo parlors, and
clothing and furniture shops. It is the most
popular destination for nightlife.

South Side Works is a recently
built, mixed-use outdoor shopping
mall, containing nationally branded apparel
stores and restaurants, upscale
condominiums, and corporate offices.

There is an small, somewhat older strip-
mall that contains the only super market
(grocery) in South Side. It also has a liquor
store, an auto-parts store, a furniture rental
store and other small chain stores.

The rest of South Side is predominantly
residential, consisting of mostly smaller row
houses.

Livehoods Found in South Side
LH8
LH9LH7
LH6
I’ll show evidence in support of the Livehoods clusters in South
Side, and will describe the forces that people highlighted.

Demographic
Differences
LH8
LH9LH7
LH6
LH8 vs LH9
“Ha! Yes! See, here is my division! Yay! Thank
you algorithm! ... I definitely feel where the
South Side Works, and all of that is, is a very
different feel.”

Architecture &
Urban Design
LH8
LH9LH7
LH6
LH7 vs LH8
“from an urban standpoint it is a lot tighter on
the western part once you get west of 17th or
18th [LH7].”

Safety
LH8
LH9LH7
LH6
LH7 vs LH8 “Whenever I was living down on 15th Street [LH7] I had
to worry about drunk people following me home, but on
23rd [LH8] I need to worry about people trying to mug
you... so it’s different. It’s not something I had
anticipated, but there is a distinct difference between the
two areas of the South Side.”

Demographic
Differences
LH8
LH9LH7
LH6
LH6 “There is this interesting mix of people there I don’t
see walking around the neighborhood. I think they
are coming to the Giant Eagle [grocery store] from
lower income neighborhoods... I always assumed
they came from up the hill.”

“I always assumed they
came from up the hill.”

Bezerkeley, CA

Other Potential Urban Analytics

• Urban Analytics

Shares your location,
gender, unique phone ID,
phone# with advertisers
Uploads your entire
contact list to their server
(including phone #s)
What are your apps really doing?

Many Smartphone Apps Have
“Unusual” Permissions
App Permissions Used
Tiny Flashlight + LED Internet Access, phone#
Backgrounds Contact List
Dictionary Location
Bible Quotes Location

Android
• What do these
permissions mean?
• Why does app need
this permission?
• When does it use
these permissions?

CrowdScanning Core Ideas
• Idea 1: find the gap between what
people expect an app to do and what
it actually does
Lin et al, Expectation and Purpose: Understanding User’s Mental
Models of Mobile App Privacy thru Crowdsourcing. Ubicomp 2012.

Nissan Maxima Gear Shift

Privacy as Expectations
• Apply this same idea of mental models
for privacy
– Compare what people expect an app
to do vs what an app actually does
– Emphasize the biggest gaps,
misconceptions that many people had
App Behavior
(What an app
actually does)
User Expectations
(What people think
the app does)

Crowdsourcing Privacy
• Idea 2: use crowdsourcing to do this
(crowdsource privacy)
• Few people read privacy policies
– We want to install the app
– Reading policies not part of main task
– Complexity of these policies (the pain!!!)
– Clear cost (time) for unclear benefit
• Crowdsourcing can mitigate these
problems

10% users were surprised this app
wrote contents to their SD card.
sent their approximate location to
dictionary.com for searching nearby
words.
sent their phone’s unique ID to
mobile ads providers.
could control their audio settings.
See all
sent their precise location to
sent their approximate location
to mobile ads providers.
sent their phone’s unique ID to
can control camera flashlight.

Our Study on App Privacy
• Showed crowd workers screenshots and
description of app (from Google Play)
– 56 of top 100 Android Apps
• Showed permissions one at a time
– Only those related to privacy
• Expectation Condition
– Why they think the app uses permission
– How comfortable they were with it
• Purpose Condition
– We gave an explanation (based on our analysis)
– Asked how comfortable they were with it

Results for Location Data
(N=20 per app, Expectations Condition)
App Comfort Level (-2 – 2)
Maps 1.52
GasBuddy 1.47
Weather Channel 1.45
Foursquare 0.95
TuneIn Radio 0.60
Evernote 0.15
Angry Birds -0.70
Brightest Flashlight Free -1.15
Toss It -1.2

Showing Purpose Lowers Concerns
• All differences statistically significant
• Big increases for dictionary, Shazam,
Air Control Lite, and others (> 1.0)
App Comfort w/
Purpose
Comfort w/o
Purpose
Device ID 0.47 ( =0.30) -0.10 ( =0.41)
Contact List 0.66 ( =0.22) 0.16 ( =0.54)
Network Location 0.90 ( =0.53) 0.65 ( =0.55)
GPS Location 0.72 ( =0.62) 0.35 ( =0.73)

Ongoing Work
• Scaling up analysis
– 600k+ apps on Android market
– Static & dynamic analysis +
clustering to build models of apps
• Ex. “Games that use location data” -1.3
• Gort tool

Summary
• Smartphones offer big opportunity
to understand human behavior at
unprecedented fidelity and scale
• Urban Analytics
• CrowdScanning

Thanks!
More info at cmuchimps.org
or email jasonh@cs.cmu.edu
Special thanks to:
• Army Research Office
• National Science Foundation
• Alfred P. Sloan Foundation
• DARPA
• Google
• CMU Cylab
Join our community for researchers at:
www.reddit.com/r/pervasivecomputing

66
Using features such as
location entropy
significantly improves
performance over
shallow features such as
number of co-locations

67

Using Location Data to
Infer Friendships
• 2.8m location sightings of
489 users of Locaccino
friend finder in Pittsburgh
• Place entropy for inferring
social quality of a place
– #unique people seen in a place
– 0.0002 x 0.0002 lat/lon grid,
~30m x 30m
Cranshaw et al, Bridging the Gap Between Physical Location and
Online Social Networks, Ubicomp 2010

• Insert graph here
• Describe entropy

Inferring Friendships
• 67 different machine learning features
– Location diversity (and entropy)
– Intensity and Duration
– Specificity (TF-IDF)
– Graph structure (overlap in friends)
• 92% accuracy in predicting friend/not
– Location entropy improves performance
over shallow features like #co-locations

Most Unexpected Uses
(N=20 per app, Expectations Condition)
• Found strong correlation between
expectations & comfort level (r=0.91)
Apps using Contact List Comfort Level (-2 – 2)
Backgrounds HD Wallpaper -1.35
Pandora -0.70
GO Launcher EX -0.75

• Insert graph here
• Describe entropy
Co-location data to infer friendship
Using place entropy, accuracy of 92%
Can also infer number of friends

Topic Modeling (LDA)

Brooklyn Queens Expressway

HCI and Smartphone Data at Scale

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (15)

Similar a HCI and Smartphone Data at Scale

Similar a HCI and Smartphone Data at Scale (20)

Último

Último (20)

HCI and Smartphone Data at Scale

Notas del editor