Web search queries for which there are no clicks are referred to as abandoned queries and are usually considered as leading to user dissatisfaction. However, there are many cases where a user may not click on any search result page (SERP) but still be satised. This scenario is referred to as good abandonment and presents a challenge for most ap- proaches measuring search satisfaction, which are usually based on clicks and dwell time. The problem is exacerbated further on mobile devices where search providers try to in- crease the likelihood of users being satised directly by the SERP. This paper proposes a solution to this problem us- ing gesture interactions, such as reading times and touch actions, as signals for dierentiating between good and bad abandonment. These signals go beyond clicks and charac- terize user behavior in cases where clicks are not needed to achieve satisfaction. We study different good abandonment scenarios and investigate the dierent elements on a SERP that may lead to good abandonment. We also present an analysis of the correlation between user gesture features and satisfaction. Finally, we use this analysis to build models to automatically identify good abandonment in mobile search achieving an accuracy of 75%, which is significantly better than considering query and session signals alone. Our fundings have implications for the study and application of user satisfaction in search systems.

1. Detecting Good Abandonment
in Mobile Search
Kyle Williams Julia Kiseleva Aidan C. Crook
Imed Zitouni Ahmed Hassan Awadallah Madian Khabsa
Pennsylvania State University
Eindhoven University of Technology
Microsoft
WWW’16, Montréal, Québec, Canada

2. Mobile Search

3. Mobile Search
• More and more popular: 2008  31% 2013  63%
• Mobile Search differs from traditional search [Human et. al, 2009]
• On Mobiles users are satisfied by the SERP [Li et. al, 2009]
• Mobiles screen is much smaller
• Mobiles are used on the way

4. Mobile Search
• More and more popular: 2008  31% 2013  63%
• Mobile Search differs from traditional search [Human et. al, 2009]
• On Mobiles users are satisfied by the SERP [Li et. al, 2009]
• Mobiles screen is much smaller
• Mobiles are used on the way
Search Engines need to adapt
And to Evaluate!

6. Knowledge Pane
Image Answer

7. Knowledge Pane
Image Answer Image Answer
Organic Results: Snippets

8. Knowledge Pane
Image Answer Image Answer
Organic Results: Snippets
Knowledge Pane

9. Evaluating User Satisfaction
• We need metrics to evaluate user satisfaction
• Good abandonment [Human et. al, 2009]:
Mobile: 36% of abandoned queries in were likely good
Desktop: 14.3%
• Traditional methods use implicit signals: clicks and dwell time

10. Evaluating User Satisfaction
• We need metrics to evaluate user satisfaction
• Good abandonment [Human et. al, 2009]:
Mobile: 36% of abandoned queries in were likely good
Desktop: 14.3%
• Traditional methods use implicit signals: clicks and dwell time
Don’t work

11. Our Main Research Problem
In the absence of clicks, what is the relationship
between a user's gestures and satisfaction and can we
use gestures to detect satisfaction and good
abandonment?

12. Research Questions
• RQ1: What SERP elements are the sources of good
abandonment in mobile search?
• RQ2: Do a user's gestures provide signals that can be used
to detect satisfaction and good abandonment in mobile
search?
• RQ3: Which user gestures provide the strongest signals for
satisfaction and good abandonment?

13. Research Questions
• RQ1: What SERP elements are the sources of good
abandonment in mobile search?
• RQ2: Do a user's gestures provide signals that can be used
to detect satisfaction and good abandonment in mobile
search?
• RQ3: Which user gestures provide the strongest signals for
satisfaction and good abandonment?
USERSTUDY

14. Research Questions
• RQ1: What SERP elements are the sources of good
abandonment in mobile search?
• RQ2: Do a user's gestures provide signals that can be used
to detect satisfaction and good abandonment in mobile
search?
• RQ3: Which user gestures provide the strongest signals for
satisfaction and good abandonment?
USERSTUDY
CROWDSOURCING

15. User Study Participants
75%
25%
GENDER
Male Female
55%
45%
LANGUAGE
English Other
82%
8%
2%
8%
EDUCATION
Computer Science Electrical Engineering
Mathematics Other
• 60 Participants
• 25.53 +/- 5.42 years

16. User Study Design
• Video Instructions (same for all participants)
• Tasks:
1. A conversion between the imperial and metric systems
2. Determining if it was a good time to phone a friend in another
part of the world
3. Finding the score from a recent game of the user’s favorite
sports team
4. Finding the user's favorite celebrity's hair color
5. Finding the CEO of a company that lost most of its value in the
last 10 years

17. Find out what is
the hair color of
your favorite
celebrity

18. Questionnaire
• Were you able to complete the task?
o Yes/No
• Where did you find the answer?
o Answer Box, Image, SERP, Visited Website
• Which query led you to finding the answer?
o First, Second, Third, >= Fourth
• How satisfied are you with your experience in this task?
o 5-point Likert scale
• Did you put in a lot of effort to complete the task?
o 5-point Likert scale

19. Questionnaire
• Were you able to complete the task?
o Yes/No
• Where did you find the answer?
o Answer Box, Image, SERP, Visited Website
• Which query led you to finding the answer?
o First, Second, Third, >= Fourth
• How satisfied are you with your experience in this task?
o 5-point Likert scale
• Did you put in a lot of effort to complete the task?
o 5-point Likert scale
5 Tasks
~20 Minutes

20. User Study Data
• Total queries – 607  563
• Abandoned queries – 576  461
• Potential abandonment tasks – 274

21. User Study Data
• Total queries – 607  563
• Abandoned queries – 576  461
• Potential abandonment tasks – 274
Binary
Labels

22. Crowdsourcing Procedure
Random sample of abandoned queries from the search logs of a
personal digital assistant during one week in June 2015 (no query
suggestion)

23. Crowdsourcing Procedure
Query: Peniston
Previous Query:
third eroics

24. Crowdsourcing Data
• Total amount of queries – 3,895
• Judgments agreement (3 per one query) – 73%
• After filtering: SAT – 1,565 and DSAT – 1,924

25. RQ1: Reasons of Good
Abandonment

26. RQ1: Reasons of Good
Abandonment
Mean of Satisfaction

27. Query and Session Features
• Session duration
• Number of queries in session
Session
Features

28. Query and Session Features
• Session duration
• Number of queries in session
• Index of query within session
• Time to next query
• Query length (number of words)
• Is this query a reformulation
• Was this query reformulated
Session
Features
Query
Features

29. Query and Session Features
• Session duration
• Number of queries in session
• Index of query within session
• Time to next query
• Query length (number of words)
• Is this query a reformulation
• Was this query reformulated
• Click count
• Number of SAT clicks (> 30 sec)
• Number of back-click clicks (< 30 sec)
Session
Features
Query
Features
Click
Features

30. Baseline 1:Click & Dwell
• Session duration
• Number of queries in session
• Index of query within session
• Time to next query
• Query length (number of words)
• Is this query a reformulation
• Was this query reformulated
• Click count
• Number of SAT clicks (> 30 sec)
• Number of back-click clicks (< 30 sec)
Session
Features
Query
Features
Click
Features
Click >
30 sec
No
Refomul
ation
B1:Click,Dwellwith
noReformulation

31. Baseline 2: Optimistic
• Session duration
• Number of queries in session
• Index of query within session
• Time to next query
• Query length (number of words)
• Is this query a reformulation
• Was this query reformulated
• Click count
• Number of SAT clicks (> 30 sec)
• Number of back-click clicks (< 30 sec)
Session
Features
Query
Features
Click
Features
NO
Click
NO
Refomul
ation
B2:Optimistic

32. Baseline 3: Query-Session Model
• Session duration
• Number of queries in session
• Index of query within session
• Time to next query
• Query length (number of words)
• Is this query a reformulation
• Was this query reformulated
• Click count
• Number of SAT clicks (> 30 sec)
• Number of back-click clicks (< 30 sec)
Session
Features
Query
Features
Click
Features
B3:Query-SessionModel:
TrainingRandomForest

33. Gesture Features (1)
• Viewport features swipes-related:
o up swipes and down swipes
o changes in swipe direction
o swiped distance in pixels and average swiped distance
o swipe distance divided by time spent on the SERP

34. Gesture Features (1)
• Viewport features swipes-related:
o up swipes and down swipes
o changes in swipe direction
o swiped distance in pixels and average swiped distance
o swipe distance divided by time spent on the SERP
• Time To Focus
o Time to focus on Answer
o Time to Focus on Organic Search Results

35. 3 seconds 6 seconds
33% of
ViewPort
66% of
ViewPort
ViewPortHeight
2 seconds
20% of
ViewPort
1s 4s 0.4s 5.4s+ + =
GF(2): Attributed Reading Time

36. 400 pixels
300 pixels
Attributed
Reading Time: 5.4s
Pixel Area:
(400 pix x 300 pix)
0.045 ms/pix2=
GF (3): Attributed Reading
Time Per Pixel

37. Models: Detecting Good Abandonment
M1: Gesture Model:
Training Random Forest based on gesture features
M2: Gesture Model + Query and Session Features:
Training Random Forest based on gesture, query and session features

38. RQ2: Are gestures useful? (1)
On only abandoned user study data:
148 SAT queries and 313 DSAT queries

39. RQ2: Are gestures useful? (2)
On crowdsourced data:
1565 SAT queries and 1924 DSAT queries

40. RQ2: Are gestures useful? (3)
On all user study data:
179 SAT queries and 384 DSAT queries
Gestures Features are useful to detect user satisfaction
in general!

41. Conclusions
• RQ1: What SERP elements are the sources of good abandonment in
mobile search?
Answer, Images and Snippet
• RQ2: Do a user's gestures provide signals that can be used to detect
satisfaction and good abandonment in mobile search?
Yes
• RQ3: Which user gestures provide the strongest signals for satisfaction
and good abandonment
Time spent interacting with Answers is positively correlated. Swipe
actions and time spent with SERP is negatively correlated

42. • Answer, Images and Snippet are
potentially source of the good
abandonment
• User gestures provide useful signals to
detect good abandonment
• Time spent interacting with Answers is
positively correlated. Swipe actions
and time spent with SERP is
negatively correlated
Questions?