Accurate models of user interest are valuable in personalizing the presentation of the often large quantity of information relevant to a query or other form of information requests. A user often interacts with multiple applications while working on a task. User models can be developed individually at each of the individual applications, but there is no easy way to come up with a more complete user model based on the distributed activity of the user. In this talk, I will introduce a novel unification framework for relevance feedback in adaptive information access; practically these models provide context for user interactions with everyday applications for user interest modeling. To tackle the cold-start problem in personalization, I will show how we can take advantage of many existing interactions combining various implicit and explicit relevance feedback indicators in a multi-application environment. I will also present a framework expanding the use of human eye movements as a source of implicit relevance feedback for user interest modeling.

1. “The Human, the Eye and the Brain : User Behavior Data Analytics“
Unifying Relevance Feedback for
User Modeling
Assistant Professor, Computer Science
Old Dominion University
1

2. User
Modeling
My Research
Oculomotor
Plant
Digital
Library
Cognition
NIRDS
Neuro-Information
Retrieval
&
Data Science
User Modeling: BigData18, IRI17, CHIIR16, JCDL15, CIKM13, SKG13
Oculomotor Plant: JCDL18, TAP13, TBME10, ETRA10, BIBE08
2

3. Outline
• Motivation : User Modeling
• Relevance Feedback in Information Search
• User Interest Models
• Eye Movements as Relevance Feedback
• Neuro-Physiological Fusion
• Consumer eye tracking
• Consumer brain tracking
3

4. Motivation
Personalized Information Retrieval
• Customize information selection and presentation
based on the inferred interest of the user
Query Input / Search History
• Most direct evidence of user
need
• But queries are short,
imprecise, and ambiguous
(e.g. synonymy and polysemy)
4

5. Relevance Feedback Categories
• Users reluctant to make frequent explicit
feedback ratings
• Rate fewer documents than they read
• Semi-explicit: Annotations and bookmarks
• Implicit: Browsing and reading behaviors
5

6. Approach: Multi-Application
“Unified” User Interest
• A unified user interest model combining
implicit and semi-explicit feedback
• A user model based on multiple everyday
applications
• A user model to predict explicit ratings from
the unified feedback
6

7. User Models
• Baseline model
text authored from production applications
• Semi-explicit model
baseline + text annotated from consumption
applications
• Unified model
semi-explicit + implicit feedback
7

8. Semi-Explicit Relevance Feedback
8
Demo : http://viewpure.com/dRIdX8kIPRQ?start=0&end=0

9. Unified Relevance Feedback
9
Sampath Jayarathna, and Frank Shipman. "Analysis and Modeling of Unified User Interests", IEEE
18th International Conference on Information Reuse and Integration , San Diego, CA, August 04-06,
2017.
Sampath Jayarathna, Atish Patra, and Frank Shipman. "Unified Relevance Feedback for Multi-
Application User Interest Modeling", Proceedings of the 15th ACM/IEEE-CS Joint Conference on
Digital Libraries, Knoxville, TN, June 22-26, 2015, pp. 129-138. (Nominated for Best Student Paper
award)
Sampath Jayarathna, Atish Patra, and Frank Shipman. "Mining User Interest from Search Tasks and
Annotations", Proceedings of ACM Conference on Information and Knowledge Management,
Burlingame, CA, October 27- November 1, 2013 , pp. 1849-1852.

10. System Architecture
10

11. Comparisons
• Chose topic modeling over clustering methods because
documents are about multiple topics
• Topic modeling applied to authored or annotated text
• LDA document-topic distribution
– Each segment is a document
– Calculate the probability that each document may contain a
topic
– Calculate similarity between document-topic distribution
– Compared with different distance metrics (H, KL, JSD)
11

12. How do the Different Similarity
Measures Perform?
Similarity Measures Performance
Precision Recall F1 Accuracy
LDA + H 0.944 0.367 0.499 0.722
LDA + KL 0.954 0.350 0.485 0.719
LDA + JCD 0.736 0.548 0.576 0.713
NMF 0.814 0.418 0.500 0.692
TF-IDF 0.247 0.396 0.287 0.237
Answer: If Recall is more important factor in generating
user models then LDA+JCD performs better than other
LDA similarity measures. For high Precision, LDA+ KL or
LDA+H perform best. 12

13. Performances
• Unified model shows significant improvement
over baseline and semi-explicit models p < 0.05
Page-Level RMSE
Task-1 Task-2 Task-3 Task-4
Baseline 1.180 1.315 1.239 1.515
Semi-Explicit 1.126 1.326 1.258 1.463
Unified 1.097 1.198 1.162 1.388
Segment-Level MAP
Baseline 0.6276 0.6371 0.6586 0.6293
Semi-Explicit 0.7827 0.6943 0.7912 0.7488
13

14. Value of “Unified” User Interest
• A unified user interest model combining
implicit and semi-explicit feedback
• Hypothesis: Unified feedback across multiple
applications results in more accurate
assessment of document value than available
through either implicit or semi-explicit
feedback alone
14

15. So Far…
• We have looked at the value of Unified
approach when locating whole resources of
interest
• aggregating activity across multiple applications for
user interest modeling
• Features described in this occasionally specific to
applications, but similar features available in most
applications involving text
• Next, we extend these approaches to other
“Sensory” feedbacks.
15

16. Big Data Aspect of User Behavior
Analytics
• Lately, the term "big data" tends to refer to the use
of predictive analytics, user behavior analytics, or certain other
advanced data analytics methods that extract value from data,
and seldom to a particular size of data set.
16

17. Big Data Aspect of User Behavior
Analytics
• Data sets grow rapidly - in part because they are increasingly
gathered by cheap and numerous information-sensing Internet
of things devices such as mobile devices, wearables, software
logs, cameras, microphones
17

18. Outline
• Motivation
• Relevance Feedback in Information Search
• User Interest Models
• Eye Movements as Relevance Feedback
• Neuro-Physiological Fusion
• Consumer eye tracking
• Consumer brain tracking
18

19. Human Oculomotor Plant Features
• Our prior work can be expanded to support a
wide range of potential applications
• This framework expands the use of eye
movements as a source of implicit relevance
feedback
• Human eye already provides a plethora of
information useful for user modeling
19

20. Eye Movements
• Eye fixations
• High acuity vision
• Eye is stable in regard to the object of interest
• Saccades
• No vision
• Move eyes between eye fixations very rapidly
• Smooth pursuits
• Various quality of vision
• Eyes follow an object
20

21. Eye Movements: Recorded Signal
21

22. Eye Movement Control
• Brain
• Oculomotor Plant
• Extraocular muscles
• eye globe with surrounding tissues
22

23. Extraocular Muscle (EOM)
KSEϴSE_M - series elasticity
component resists
contractile force
KLTϴLT_M - additional
contractile force is added to
active state tension by
length-tension component
FM - active state tension –
contracts muscle
NM - neuronal control
signal sent by brain 23

24. Agonist Extraocular Muscle
BAG.V - damping
component (force velocity
relationship) resists a
muscle shortening
24

25. 2D Oculomotor Plant (2DOPMM)
25

26. 2D Oculomotor Plant (2DOPMM)
• 12 Differential equations
• 36 OP Features
• The model can simulate saccades with characteristics
of “normal” humans
26

27. Can OPF Reveal Relevance?
Classified
Saccades
Simulated
Saccades
Error
Function
Relevance
Prediction
27

28. OPF-Aware User Model
28

29. OPF-Aware User Model
29

30. Saccade-Bounded OPF
• The motivation behind out work is that not all
gazes are created equal.
• Fixations require preceding saccades help to
place the gaze in the stimuli to gather
information from the target location.
30
Sampath Jayarathna and Sobiga Shanmugathasan. " Evaluating Saccade-bounded Eye Movement
Features for the User Interest Modeling", Proceedings of the 18th ACM/IEEE-CS Joint Conference
on Digital Libraries, Fort Worth, TX, June 3-6, 2018. [In press]
Sampath Jayarathna, and Frank Shipman. "Rationale and Architecture for Incorporating Human
Oculomotor Plant Features in User Interest Modeling", Proceedings of the ACM Conference on
Human Information Interaction and Retrieval, Chapel Hill, NC, March 13-17, 2016, pp.281-284.

31. Saccade-Bounded OPF
31

32. Neuro Physiological Feature Fusion
32

33. Consumer Eye Tracking!
Tobii x120
60 Hz/120 HZ, 0.5 degree
$45K
Tobii EyeX and EyeTribe
30-75 HZ, 0.5-1.0 degree
$99 / $140
Pupil Labs
200 Hz
$2500
33

34. Consumer Brain EEG!
•Emotive Epoc 14 Channel EEG
•Designed for gaming,
•consumer biofeedback
•Wireless, portable
•Affordable
–$300 basic headset, Emotiv Insight
–$750 research, Emotiv epoc+
– $2000 research, Emotiv Flex 34

35. Consumer/Research Wearables!
•Empatica E4 $1800
35

36. Thanks to my
Collaborators/Students
36

37. Thank you!
37
• Web : https://www.cs.odu.edu/~sampath/
• Contact : sampath@cs.odu.edu
• WS-DL : https://ws-dl.cs.odu.edu/
• Twitter : @OpenMaze @WebSciDL
Questions and Comments