In recent years, the use of Artificial Intelligence (AI) technologies expanded to many areas where they directly affect the lives of many people. AI-based approaches advise human decision-makers who should be released on bail, whether it is a good time to discharge a patient from a hospital and whether a specific student is at risk to fail a course. Such an extensive use in AI in decision making came with a range of protentional problems that have been extensively studied over the last few years. Recognition of these problems motivated a rapid rise of research on “human-centered AI”, which attempted to address and minimize the negative effects of using AI technologies. Among the ideas of human-centered AI is user control - engaging users in affecting AI decision making to prevent possible errors and biases. In my talk, I will focus on the application of user control in one popular area of AI application, adaptive information access. Adaptive information access systems such as personalized search and recommender systems attempt to model their users to help them in finding the most relevant information. Yet, user modeling and personalization mechanisms might not always work as expected resulting in errors, biases, and suboptimal behavior. Combining the decision power or AI with the ability of the user to guide and control it brings together the strong sides of artificial and human intelligence and could lead to better results. In my talk, I review several projects focused on user control in adaptive information access systems and discuss the benefits and challenges of this approach.

1. Two Brains are Better than One:
User Control in Adaptive
Information Access
Peter Brusilovsky
with Jae-Wook Ahn, Denis Parra,
Katrien Verbert, Chun-Hua Tsai
PAWS Lab
School of Computing and Information
University of Pittsburgh

2. AI or Humans + AI?
3

3. Two Brains are Better than One!
4
Imagecredit:https://towardsdatascience.com

4. Adaptive Information Access
Adaptive
Hypermedia
Adaptive
IR
Recommender
Systems
Navigation Search Recommendation
Metadata-based
mechanism
Keyword-based
mechanism
Community-
based mechanism
Adaptation Mechanisms
Types of information access

5. ADAPTIVE HYPERMEDIA
Adding AI to user-controlled information access environment
7

6. Navigation vs. Adaptive Sequencing
8

7. Adaptive Navigation Support

8. ELM-ART: Adaptive Annotation (1996)
Weber,G.andBrusilovsky,P.(2001)ELM-ART:Anadaptiveversatilesystemfor
Web-basedinstruction.InternationalJournalofArtificialIntelligencein
Education12(4),351-384.

9. NavEx: Adaptive Annotation
Brusilovsky,P.andYudelson,M.(2008)FromWebExtoNavEx:InteractiveAccess
toAnnotatedProgramExamples.ProceedingsoftheIEEE96(6),990-999.

10. Adaptive Annotation Can:
• Reduce navigation efforts
• Reduce repetitive visits to learning content
pages
• Encourage non-sequential navigation
• Increase learning outcome
• For those who is ready to follow and advice
• Make system more attractive for students
• Students stay much longer without any reward

11. ELM-ART: Evaluation
• No formal classroom study
• Users provided their experience
• Drop-out evaluation technology
• 33 subjects
– visited more than 5 pages
– have no experience with Lisp
– did not finish lesson 3
– 14/19 with/without programming

12. ELM-ART: Value of ANS
Mean number of pages which the users with no experience in
programming languages completed with ELM-ART

13. ELM-ART: Value of ANS
Mean number of pages which the users with experience in at
least one programming language completed with ELM-ART

14. USER-CONTROLLED
PERSONALIZATION
Adding user-control to AI-driven information access in
personalized search and recommender systems
16

15. • Compromise between several sources of relevance
– Items might be relevant for to the user profile or query
for different reasons
• Single-source: different parts/aspects of the profile
• Hybrid: different sources of information or approaches
• Hard to get universally perfect ranking
– A recommendation approach is tuned to an
overall/generic situation, but users could consult
recommendation for different needs
– Some profile aspects, sources, approaches are less
relevant in the current context, but some are more
17
While Single Ranked List is A Problem?

16. 18

17. What are Possible Solutions?
• Control (Keep the ranked list, better engage users)
– Change user profile
– Change parameters (how personalization is produced)
• Visualize and Explore (Go beyond the ranked list)
– Present items visually
– Make the ranking/relevance process more transparent
– Allow users to change presentation parameters, play
with the results, better understand the process, isolate
most relevant results
19

18. CONTROL!
Allow the user to control multiple aspects of the recommendation
process to better adapt personalization for the current context as
well as better explore recommendation results
20

19. What Can Be Controlled?
21
Profile Generation Presentation
User Model Single Source
Fusion
EXPLORE!

20. Open Learner Model (ELM-ART)
22Weber, G. and Brusilovsky, P. (2001) ELM-ART: An adaptive versatile system for Web-based instruction. International Journal of
Artificial Intelligence in Education 12 (4), 351-384.

21. Open User Model (YourNews)
Ahn, J.-w., Brusilovsky, P., Grady, J., He, D., and Syn, S. Y. (2007) Open user profiles for adaptive news systems: help or harm? In: 16th
international conference on World Wide Web, WWW '07, Banff, Canada, May 8-12, 2007, ACM, pp. 11-20

22. Concept-Level Open User Model
(SciNet)
24
Glowacka, Dorota, Tuukka Ruotsalo, Ksenia Konuyshkova, Kumaripaba Athukorala, Samuel Kaski, and Giulio Jacucci. 2013.
"Directing Exploratory Search: Reinforcement Learning from User Interactions with Keywords." In international conference on
Intelligent user interfaces, IUI '2013, 117-27. Santa Monica, USA: ACM Press.

23. TaskSieve: Controllable Personalized Search
Ahn, Jae-wook, Peter Brusilovsky, Daqing He, Jonathan Grady, and Qi Li. 2008. "Personalized Web Exploration with Task Models." In the
17th international conference on World Wide Web, WWW '08, 1-10. Beijing, China: ACM.

24. TaskSieve Controllable Ranking
• Post-filtering
• Combine query relevance and task relevance
– Alpha * Task_Model_Score + (1-alpha) * Search Score
– Alpha : user control (0.0, 0.5, or 1.0)
• Results
– Better than regular adaptive search
– Better then non adaptive baseline even in cases when
profile was excluded
– Users were really good in deciding when to engage the
profile and how
26

25. O'Donovan, John, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. "PeerChooser: visual interactive recommendation."
In Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, 1085-88. Florence, Italy: ACM.
PeerChooser: Controllable CF
27

26. EXPLORE!
Make the ranking process transparent and explorable. Allow users
to play with presentation parameters to understand aspects of
relevance and find best items in the given context
28

27. Control and Transparency:
Two Sides of the Same Coin
Explain Visualize
ExploreControl
29
Transparency
Controllability
No full transparency
without controllability
Control is challenging
without transparency

28. TasteWeights: Profile and Mechanism
Control
30
Knijnenburg, Bart P., Svetlin Bostandjiev, John O'Donovan, and Alfred Kobsa. 2012. "Inspectability and Control in Social Recommenders." In 6th ACM
Conference on Recommender System, 43-50. Dublin, Ireland.

29. Multiple Sources of Relevance
• Conference Navigator System for conference support (2010+)
• Classic content-based relevance prospects (search)
– Items that has a specific keyword
• Social relevance prospects (browsing)
– Items bookmarked by a socially connected user
• Tag relevance prospects (browsing)
– Items tagged by a specific tag
• Personal relevance prospects (recommendation)
– Several different recommender engines
– Each engine offer one relevance prospect
31
Brusilovsky, P., Oh, J. S., López, C., Parra, D., and Jeng, W. (2017) Linking information and people in a social system for academic
conferences. New Review of Hypermedia and Multimedia 23 (2), 81-111.

30. SetFusion: User-Controlled Fusion
• Using set relevance visualization in
the familiar Venn diagram form
– One recommendation source = one set
• Allow controlled ranking
fusion
• Combine ranking with
annotation showing source(s)
of recommendation
36
Parra, D. and Brusilovsky, P. (2015) User-controllable personalization: A case study
with SetFusion. International Journal of Human-Computer Studies 78, 43–67.

32. Brief Results of Two Studies
• SetFusion provides strong engaging effect
– Number of engaged users, bookmarked talks,
explored talks doubled
– The effect is larger in UMAP “natural” settings
• SetFusion allows more efficient work
– Increases yield of bookmarks in relation to
overhead actions
• But only 3 dimensions of relevance with Venn!
• How to control for more than 3 dimensions?
43

33. RelevanceTuner: Control+Visualization
in a Hybrid Social Recommender
Tsai, Chun-Hua and Peter Brusilovsky (2018) Beyond the Ranked List: User-Driven Exploration and Diversification of Social
Recommendation. In 23rd International Conference on Intelligent User Interfaces, 239--50. Tokyo, Japan: ACM.

34. VISUALIZE+EXPLORE!
Present recommendations visually helping users to understand
how relevance mechanism work
46

35. Experiments with Visual
Exploration
• Adaptive Vibe (2006-2015)
– With Jae-Wook Ahn
• Relevance Explorer (2013-2016)
– With Katrien Verbert and Denis Parra
• Intersection Explorer (2017-2019)
– With Katrien Verbert, Karsten Seipp, Chen He, Denis
Parra, Bruno Cardoso, Gayane Sedrakyan, Francisco
Gutiérrez
• ScatterViz (2018)
– With Chun Hua Tsai
47

36. Adaptive VIBE: Exploring and
Controlling Adaptive Search
49https://www.youtube.com/watch?v=Yt1fMEFlLVA&index=2&list=PLyCV9FE42dl7JG_i7m_kvwuYRpfwwJ4iY
Ahn,Jaewook,andPeterBrusilovsky.2013.'Adaptivevisualizationfor
exploratoryinformationretrieval',InformationProcessingandManagement,
49:1139–64.

38. VIBE based query-profile fusion
User Profile Terms
Query Terms
Documents
Mixing user profile and query terms as VIBE POI

39. • User profile is added on the same playfield
as user query
• Topology is adaptive
• Mediate between profile (green POI) and
query (red POI) terms
• Browse documents free with control on
profile and query terms
Adaptive topology in VIBE

40. Adaptive VIBE with Concepts
56

41. Some Study Results
• A sequence of user studies
– Search vs. VIBE vs. VIBE+NE
• Search -> VIBE -> VIBE+NE offers:
– Better visual separation of relevant documents (system)
– Supports better opening relevant documents (user)
• VIBE+NE supports more meanigful interaction
– No degradation found even with active visual UM
manipulation
– While over performance retained or increased
Ahn, J., Brusilovsky, P., and Han, S. (2015) Personalized Search: Reconsidering the Value of Open User Models. In:
Proceedings of Proceedings of the 20th International Conference on Intelligent User Interfaces, Atlanta, Georgia, USA, March 29-
April 1, 2015, ACM, pp. 202-212

42. Relevance Explorer
• Context: multiple dimensions of relevance
– social - users, content - tags, recommender engines
• Using set relevance visualization
– One dimension of relevance = one set
• Agent metaphor to mix user- tag- and
engine-based relevance
– Users, tags, and recommender systems are shown as
agents collecting relevant talks
– Multiple-relevance match -> stronger evidence
59

43. TalkExplorer
• Recommendation engines are shown as agents in parallel to users and tags
• Uses Aduna clustermap library: http://www.aduna-software.com/
60

44. Interrelations agents and users
64

45. Evaluation
• Setup
– supervised user study
– 21 participants at UMAP 2012 and ACM Hypertext 2012 conferences
• Results
– The more aspects of relevance are fused, the more effective it is for
getting to relevant items. Especially effective are fusions across
relevance dimensions
– The more relevance prospects are merged, the better is the yield, the
easier is to find good items
– Dimensions of relevance are not equal
– ADUNA approach is challenging for beyond fusion of 3 aspects 65
Verbert, K., Parra-Santander, D., and Brusilovsky, P. (2016) Agents Vs. Users: Visual Recommendation of Research Talks
with Multiple Dimension of Relevance. ACM Transactions on Interactive Intelligent Systems 6 (2), Article No. 11

46. Intersection Explorer
• Based on ideas of
SetFusion and Talk
Explorer
• New approach for
scalable multi-set
visualization
66
Cardoso, Bruno, Gayane Sedrakyan, Francisco Gutiérrez, Denis Parra, Peter Brusilovsky, and Katrien Verbert. 2019. 'IntersectionExplorer, a multi-
perspective approach for exploring recommendations', International Journal of Human-Computer Studies, 121: 73-92.

47. Intersection Explorer (2017)
67

48. ScatterViz: Diversity-Focused
Exploration of Hybrid Recommendations
Tsai, Chun-Hua, and Peter Brusilovsky. 2018. "Beyond the Ranked List: User-Driven Exploration and Diversification of Social
Recommendation." In 23rd International Conference on Intelligent User Interfaces, 239--50. Tokyo, Japan: ACM.

49. Questions?
69

50. Readings
• Ahn, Jae-wook, Peter Brusilovsky, Jonathan Grady, Daqing He, and Sue Yeon Syn (2007) Open user profiles
for adaptive news systems: help or harm? In the 16th international conference on World Wide Web, WWW '07, 11-20.
• Ahn, Jae-wook, Peter Brusilovsky, Daqing He, Jonathan Grady, and Qi Li.( 2008.) Personalized Web
Exploration with Task Models."In the 17th international conference on World Wide Web, WWW '08, 1-10. Beijing, China:.
• Ahn, J. and Brusilovsky, P. (2013) Adaptive visualization for exploratory information retrieval. Information Processing
and Management 49 (5), 1139–1164.
• Ahn, J., Brusilovsky, P., and Han, S. (2015) Personalized Search: Reconsidering the Value of Open User Models. In:
Proceedings of Proceedings of the 20th International Conference on Intelligent User Interfaces, Atlanta, Georgia, USA,
March 29-April 1, 2015, ACM, pp. 202-212
• Verbert, K., Parra-Santander, D., and Brusilovsky, P. (2016) Agents Vs. Users: Visual Recommendation of
Research Talks with Multiple Dimension of Relevance. ACM Transactions on Interactive Intelligent Systems 6 (2), Article
No. 11
• Parra, D. and Brusilovsky, P. (2015) User-controllable personalization: A case study with SetFusion. International
Journal of Human-Computer Studies 78, 43–67.
• Cardoso, Bruno, Gayane Sedrakyan, Francisco Gutiérrez, Denis Parra, Peter Brusilovsky, and Katrien
Verbert (2019). IntersectionExplorer, a multi-perspective approach for exploring recommendations, International
Journal of Human-Computer Studies, 121: 73-92.
• Verbert, K., Parra-Santander, D., Brusilovsky, P., Cardoso, B., and Wongchokprasitti, C. (2017) Supporting
Conference Attendees with Visual Decision Making Interfaces. In: Companion of the 22nd International Conference on
Intelligent User Interfaces (IUI '17), Limassol, Cyprus, ACM.
• Tsai, Chun-Hua and Peter Brusilovsky (2018) Beyond the Ranked List: User-Driven Exploration and Diversification
of Social Recommendation. In 23rd International Conference on Intelligent User Interfaces, 239--50. Tokyo, Japan: ACM.
70