Presentation at CIKM, held in San Francisco, October 2013. Users often access and re-access more than one site during an online session, effectively engaging in multitasking. In this work, we study the effect of online multitasking on two widely used engagement metrics designed to capture users browsing behavior with a site. Our study is based on browsing data of 2.5M users across 760 sites encompassing diverse types of services such as social media, news and mail. To account for multitasking we need to redefine how user sessions are represented and we need to adapt the metrics under study. We introduce a new representation of user sessions: tree-streams – as opposed to the commonly used click-streams – present a more accurate picture of the browsing behavior of a user that includes how users switch between sites (e.g., hyperlinking, teleporting, backpaging). We then discuss a number of insights on multitasking patterns, and show how these help to better understand how users engage with sites. Finally, we define metrics that characterize multitasking during online sessions and show how they provide additional insights to standard engagement metrics. This work was done in collaboration with Mounia Lalmas, George Dupret, and Ricardo Baeza-Yates.

1. ONLINE MULTITASKING
AND USER ENGAGEMENT
CIKM 2013
In
collabora*on
with:
Mounia
Lalmas,
Ricardo
Baeza-­‐Yates,
George
Dupret
Jane%e
Lehmann

2. outline
1. Mo%va%on
How
do
users
browse
the
web
today?
2. Characteris%cs
of
online
mul%tasking
Ac2vity
during
and
between
visits
3. Measuring
online
mul%tasking
Defini2on
of
new
metrics,
case
study
Lights
on
by
JC*+A!

3. How
do
users
browse
the
Web
today?
leC
by
[
embr
]

4. ONLINE MULTITASKING
4
JaneGe
Lehmann
Mo2va2on
Browsing
the
“old
way”
facebook
news
news
news
news
mail
1min
2min
1min
3min
Dwell
2me
during
a
visit
on
a
news
site:
7min
on
average
news
site

5. ONLINE MULTITASKING
5
JaneGe
Lehmann
Mo2va2on
Nowadays
news
facebook
mail
news
news
news
1min
2min
1min
3min
Dwell
2me
during
a
visit
on
a
news
site:
2.33min
on
average
(1min
|
3min
|
3min)

6. ONLINE MULTITASKING
6
JaneGe
Lehmann
Mo2va2on
• Users
switch
between
sites,
to
do
related
or
totally
unrelated
tasks
• E.
Herder
[1]:
» 75%
of
sites
are
visited
more
than
once
» 74%
of
revisits
are
performed
within
a
session
Measuring
browsing
behavior
can
lead
to
incorrect
conclusions.
[1]
E.
Herder.
Characteriza*ons
of
user
web
revisit
behavior.
In
LWA,
2005.

7. Characteris%cs
of
online
mul%tasking
Danboard's
Messy
Home
by
Mullenkedheim

8. DATA SET
Interac%on
data
• July
2012
• 2.5M
users
• 785M
page
views
• We
defined
a
new
naviga2on
model
(see
paper
for
detail)
• Categoriza2on
of
the
most
frequent
accessed
sites
(e.g.
mail,
news,
shopping)
» 11
categories
(news),
33
subcategories
(e.g.
news
finance,
news
society)
» 760
sites
from
70
countries/regions
8
JaneGe
Lehmann
Characteris2cs

9. Visit activity
Visit
frequency
9
JaneGe
Lehmann
Characteris2cs
Mul%tasking
depends
on
the
site
under
considera%on
• Social
media
sites
are
revisited
the
most
• News
(tech)
sites
are
the
least
revisited
sites
news (finance)
news (tech)
social media
mail
2.09
1.76
2.28
2.09
4.65
1.59
4.78
4.61
#Visits
(avg sd)

10. Visit activity
Ac%vity
between
visits
10
JaneGe
Lehmann
Characteris2cs
Differences
in
the
absence
%me
• 50%
of
sites
are
revisited
aCer
less
than
1min
-­‐
Interrup*on
of
a
task
• There
are
revisits
aCer
a
long
break
-­‐
Returning
to
a
site
to
perform
a
new
task
0.00
0.25
0.50
0.75
1.00
10 2
10 1
100
101
102
mail
social media
news (finance)
news (tech)
Cumulativeprobability
Absence time [min]
*
v2
v1
*
v3
*
-­‐
absence
2me

11. Visit activity
Ac%vity
paLern
11
JaneGe
Lehmann
Characteris2cs
• Four
types
of
"aGen2on
shiCs”
• Complex
cases
refer
to
no
specific
paGern
or
repeated
paGern
• Successive
visits
can
belong
together
(i.e.,
to
the
same
task)
0.23
0.28
0.33
mail sites
news (finance) sites news (tech) sites
social media sites
decreasing attention increasing attention
constant attention complex attention
Proportionoftotal
dwelltimeonsite
p-value = 0.09
m = -0.01
p-value = 0.07
m = -0.02
p-value = 0.79
m = 0.00
0.23
0.28
0.33
Proportionoftotal
dwelltimeonsite

12. Danboard
by
sⓘndy°
Measuring
online
mul%tasking

13. Cumulative activity
Cumula%ve
ac%vity
vi
Browsing
ac2vity
during
the
ith
visit
ivi
Browsing
ac2vity
between
the
(i-­‐1)th
and
ith
visit
k=3
Rescaling
factor
for
ivi
m
Browsing
ac2vity
(e.g.
dwell
2me,
page
views)
Assump%on:
If
users
return
aCer
short
2me,
they
return
to
con2nue
with
same
task.
If
users
return
aCer
longer
2me,
they
return
to
perform
a
new
task
-­‐
an
indica2on
of
loyalty
to
the
site.
13
JaneGe
Lehmann
Metrics
CumActm,k = log10 (v1 + ivi
k
•vi
i=2
n
∑ )
iv2
v2
v1
iv3
v3

14. Cumulative activity
Cumula%ve
ac%vity
vi
Browsing
ac2vity
during
the
ith
visit
ivi
Browsing
ac2vity
between
the
(i-­‐1)th
and
ith
visit
k=3
Rescaling
factor
for
ivi
m
Browsing
ac2vity
(e.g.
dwell
2me,
page
views)
Assump%on:
If
users
return
aCer
short
2me,
they
return
to
con2nue
with
same
task.
If
users
return
aCer
longer
2me,
they
return
to
perform
a
new
task
-­‐
an
indica2on
of
loyalty
to
the
site.
14
JaneGe
Lehmann
Metrics
CumActm,k = log10 (v1 + ivi
k
•vi
i=2
n
∑ )
iv2
v2
v1
iv3
v3

15. Cumulative activity
Cumula%ve
ac%vity
vi
Browsing
ac2vity
during
the
ith
visit
ivi
Browsing
ac2vity
between
the
(i-­‐1)th
and
ith
visit
k=3
Rescaling
factor
for
ivi
m
Browsing
ac2vity
(e.g.
dwell
2me,
page
views)
Assump%on:
If
users
return
aCer
short
2me,
they
return
to
con2nue
with
same
task.
If
users
return
aCer
longer
2me,
they
return
to
perform
a
new
task
-­‐
an
indica2on
of
loyalty
to
the
site.
15
JaneGe
Lehmann
Metrics
CumActm,k = log10 (v1 + ivi
k
•vi
i=2
n
∑ )
iv2
v2
v1
iv3
v3
v1
+
v2
+
v3

16. Cumulative activity
Cumula%ve
ac%vity
vi
Browsing
ac2vity
during
the
ith
visit
ivi
Browsing
ac2vity
between
the
(i-­‐1)th
and
ith
visit
k=3
Rescaling
factor
for
ivi
m
Browsing
ac2vity
(e.g.
dwell
2me,
page
views)
Assump%on:
If
users
return
aCer
short
2me,
they
return
to
con2nue
with
same
task.
If
users
return
aCer
longer
2me,
they
return
to
perform
a
new
task
-­‐
an
indica2on
of
loyalty
to
the
site.
16
JaneGe
Lehmann
Metrics
CumActm,k = log10 (v1 + ivi
k
•vi
i=2
n
∑ )
iv2
v2
v1
iv3
v3

17. Cumulative activity
Cumula%ve
ac%vity
vi
Browsing
ac2vity
during
the
ith
visit
ivi
Browsing
ac2vity
between
the
(i-­‐1)th
and
ith
visit
k=3
Rescaling
factor
for
ivi
m
Browsing
ac2vity
(e.g.
dwell
2me,
page
views)
Assump%on:
If
users
return
aCer
short
2me,
they
return
to
con2nue
with
same
task.
If
users
return
aCer
longer
2me,
they
return
to
perform
a
new
task
-­‐
an
indica2on
of
loyalty
to
the
site.
17
JaneGe
Lehmann
Metrics
CumActm,k = log10 (v1 + ivi
k
•vi
i=2
n
∑ )
iv2
v2
v1
iv3
v3
v1
+
(iv2)3Ÿ
v2
+
(iv3)3Ÿ
v3

18. Activity pattern
ALen%on
shiN
and
range
n=4
Number
of
visits
in
session
σ
Variance
in
the
visit
ac2vity
μ
Average
of
the
visit
ac2vity
inv
Modifica2on
of
the
“Inversion
number”
Descrip%on:
AGShiC
models
the
shiC
of
aGen2on
in
the
browsing
ac2vity
AGRange
describes
fluctua2ons
in
the
browsing
ac2vity
18
JaneGe
Lehmann
Metrics
AttShiftm,n =
invm,n − minInvm,n
| maxInvm,n |− | minInvm,n |
AttRangem,n =
σ (Vm,n )
µ(Vm,n )

19. Activity pattern
ALen%on
shiN
and
range
19
JaneGe
Lehmann
Metrics
-­‐1
0
1
0
constant
constant
constant
>
0
decreasing
complex
increasing
AUen*on
shiV
AUen*on
range

20. Comparing
the
ranking
of
the
sites
• Visitdt
–
Dwell
2me
during
a
visit
• Sessiondt
–
Dwell
2me
during
a
session
Ø Visitdt
and
Sessiondt
correlate
Ø Otherwise
no
correla2on
à
the
other
metrics
capture
different
aspects
of
browsing
behavior
Comparing metrics
20
JaneGe
Lehmann
Metrics
Visitdt
Sessiondt
CumActdt
ALShiNdt
Sessiondt
0.57
CumActdt
-­‐0.04
0.24
ALShiNdt
0.09
0.22
0.02
ALRangedt
-­‐0.01
-­‐0.01
-­‐0.26
0.19

21. “Models”
of
browsing
behavior
• Clustering
of
sites
using
mul2tasking
and
standard
engagement
metrics:
• CumActdt,
AGShiCdt,
AGRangedt
• Visitdt,
Sessiondt
• We
iden2fied
five
cluster:
Models of browsing behavior
21
JaneGe
Lehmann
Metrics
C4: 74 sites
0.25
-0.25
0.75
-0.75
C5: 166 sites
0.25
-0.25
0.75
-0.75
C3: 156 sites
0.25
-0.25
0.75
-0.75
C2: 108 sites
0.25
-0.25
0.75
-0.75
C1: 172 sites
0.25
-0.25
0.75
-0.75
Visitdt
[min] CumActdt,3
AttShiftdt,4
AttRangedt,4
Sessiondt
[min]

22. Models of browsing behavior
22
JaneGe
Lehmann
Metrics
Visitdt
[min] CumActdt,3
AttShiftdt,4
AttRangedt,4
Sessiondt
[min]
C2: 108 sites
auctions, front page,
shopping, dating
0.25
-0.25
0.75
-0.75
C1: 172 sites
mail, maps, news,
news (soc.)
0.25
-0.25
0.75
-0.75
One
task
during
a
session
§ High
dwell
2me
per
visit
and
during
the
whole
session
§ Users
return
to
con2nue
a
task
(short
absence
2me)
§ C1:
aGen2on
is
shiCing
to
another
site
§ C2:
aGen2on
is
shiCing
slowly
towards
the
site

23. C4: 74 sites
front page, search,
download
C3: 156 sites
auctions, search,
front page, shopping
0.25
-0.25
0.75
-0.75
0.25
-0.25
0.75
-0.75
Models of browsing behavior
23
JaneGe
Lehmann
Metrics
Several
tasks
during
a
session
§ Users
perform
several
tasks
on
these
sites
during
a
session
§ No
simple
ac2vity
paGern
§ C3:
Dwell
2me
per
visit
is
low,
but
the
dwell
2me
per
session
is
high
Visitdt
[min] CumActdt,3
AttShiftdt,4
AttRangedt,4
Sessiondt
[min]

24. C5: 166 sites
service, download,
blogging, news (soc.)
0.25
-0.25
0.75
-0.75
Models of browsing behavior
24
JaneGe
Lehmann
Metrics
Sites
with
low
ac%vity
§ Users
do
not
spend
a
lot
of
2me
on
these
sites
§ Time
between
visits
is
short
§ AGen2on
is
shiCing
towards
the
site
Visitdt
[min] CumActdt,3
AttShiftdt,4
AttRangedt,4
Sessiondt
[min]

25. C2: 108 sites
auctions, front page,
shopping, dating
0.25
-0.25
0.75
-0.75
C3: 156 sites
auctions, search,
front page, shopping
0.25
-0.25
0.75
-0.75
Models of browsing behavior
25
JaneGe
Lehmann
Metrics
Browsing
behavior
can
differ
between
sites
of
the
same
category
§ C2:
users
visit
site
once
to
perform
their
task
§ C3:
users
visit
site
several
2mes
to
perform
task(s)
Visitdt
[min] CumActdt,3
AttShiftdt,4
AttRangedt,4
Sessiondt
[min]

26. SUMMARY and Future Work
JaneGe
Lehmann
26
• Online
mul2tasking
affects
the
way
users
access
sites
–
Standard
metrics
do
not
capture
this!!!
• We
defined
metrics
that
describe
different
aspects
of
mul2tasking
• CumAct
accounts
for
the
2me
between
visits
• AGShiC,
AGRange
describe
aGen2on
shiCs
• We
showed
that
mul2tasking
depends
on
the
site
under
considera2on
Future
work:
• Can
we
improve
the
defini2on
of
a
task?
• How
does
mul2tasking
affect
other
metrics,
such
as
bounce
rate
and
click-­‐
through
rate?
• Does
mul2tasking
differ
in
different
countries?
Summary

27. Janette Lehmann
Universitat Pompeu Fabra, Spain
lehmannj@acm.org
Mounia Lalmas
Yahoo Labs London
mounia@acm.org
George Dupret
Yahoo Labs Sunnyvale
gdupret@yahoo-inc.com
Ricardo Baeza-Yates
Yahoo Labs Barcelona
rbaeza@acm.org
Online
Multitasking
+
User
Engagement