1.
A Task-based Scientific
Paper Recommender
System for Literature
Review and Manuscript
Preparation
Aravind SESAGIRI RAAMKUMAR
PhD Candidate
Oral Examination Presentation for fulfillment of
PhD
October 3rd 2017

2.
Citation
recommendations
Research
paper
recommender
systems
Recommending
papers
{Ad-hoc search}
Background
“How to get the best set of relevant documents for a researcher’s literature review and
publication purposes?”
SPRS
Research
Area
More than
250 papers
Process-based
Interventions
Technology-oriented
Interventions
2

3.
Identify
research
opportunities
Find
collaborators
Secure
support
Review the
literature
Collect
research data
Analyze
research data
Disseminate
findings
Manage the
research
process
Related Work Classification (1/2)
Active and Explicit
Information Needs
(AEIN) in the
Research Lifecycle
A) Active and Explicit Information Needs (AEIN)
B) Passive and Implicit Information Needs (PIIN)
<Recommendation of Scholarly Objects>
• Building a reading list for
literature review
• Finding similar papers
:
:
:
• Searching papers based
on input text
• Publication Venues
• Citation Context
3

4.
Related Work Classification (2/2)
Passive and Implicit Information Needs (PIIN)
• User Footprint
• Researcher’s Publication History
• Social Network of Authors
• Social Tags
• Reference Management Systems
4

5.
Research Gaps in SPRS Studies
Consolidated
Framework for
Contextual
Dimensions
Lack of
Connectivity
between Tasks
Lack of
Relation(s)
between Tasks
and RS Filtering
Mechanisms
Absence of
Article Type as
an Input
Dimension
5

6.
Research Objectives
• To identify an appropriate method to map the identified LR and
MP tasks to relevant IR/RS algorithms
– RQ1: What are the key search tasks of researchers in the
literature review and publication lifecycle?
– RQ2: How to relate the identified tasks of researchers to
IR/RS algorithms?
• To evaluate whether the performance of the proposed
recommendation techniques for the tasks and the overall
system were at the expected level
– RQ3: Do the proposed recommendation techniques of the
relevant tasks outperform the existing baseline
approaches in system-based evaluation?
– RQ4: Do the proposed recommendation techniques and
the overall system meet the expected standards in user-
based evaluation?
Study I
Study II
Rec4LRW
System
6

7.
Study I - Survey on Inadequate and Omitted
Citations (IOC) in Manuscripts
Authors
Reviewers
Problems
in research
quality
Manuscripts with
improper LR
7

8.
Study I - Survey on Inadequate and Omitted
Citations (IOC) in Manuscripts
Aims
• What are the critical instances of IOC?
• Do the critical instances and reasons of IOC in research manuscripts relate
with the scenarios/tasks where researchers need external assistance in
finding papers?
• Identify the prominent information sources
• What is the researchers’ awareness level of available recommendation
services for research papers?
8

9.
Study I Details
• Single center data collection conducted for two months
• Only researchers with paper authoring experience were recruited
• 207 NTU researchers participated in the study
71% of the participants answered from both reviewer and author perspectives
• Survey questionnaire comprising of 31 questions
• Agreeability measured in 5-point Likert scale
• Data analyses through one-sample t-test with test value of either
2 or 3
9

10.
Study I - Results
Instances of IOC
• Authors viewpoint
– Missed citing seminal and topically-similar papers in
journal manuscripts
• Reviewers viewpoint
– Missed citing seminal, topically-similar papers in all
manuscripts
– Insufficient and irrelevant papers in the LR of all
manuscripts
Effects of IOC
• Reviewers viewpoint
– Manuscripts are sent back for revision due to missing
citations
10

11.
Study I - Results
Need for External Assistance in Finding Papers
• Authors required support for the below papers:-
1. Interdisciplinary papers
2. Topically-similar papers
3. Seminal papers
4. Citations for placeholders in manuscripts
5. Necessary citations meant for inclusion in manuscripts
Usage of Academic Information Sources
• Researchers used the below sources in the order of usage
1. Google Scholar
2. ScienceDirect
3. Web of Science
4. SpringerLink
• 62% of the participants have never used SPRS services
11

12.
Study I – Key Findings
• Researchers need help in finding interdisciplinary, topically-
similar and seminal papers
• Generating reading list (seminal papers) and finding similar
papers are two necessary LR search tasks for the
proposed system
• Shortlisting papers from final reading list for inclusion in
manuscript, selected as third task for the proposed system
• Google Scholar’s simplistic UI makes it the most used
information source and ideal choice for UI design of a new
assistive system
12

13.
Rec4LRW Design and Development
Task Redesign
Task
Interconnectivity
Informational
Display Features
13

14.
Rec4LRW System Design - I
Base Features
• Plug and Play concept
• Features represent different characteristic of paper and its relations to references and citations
• Grey Literature Percentage, Coverage, Textual Similarity and Specificity are novel features
• New features can be added as required
14

15.
Rec4LRW System Design - II
Task 1 -Building an Initial Reading List of Research Papers
Popular
papers
Recent
papers
Survey
papers
Diverse
papers
Use of Okapi BM25 Similarity Score to retrieve
top 200 matching papers
Requirements
Author-specified Keywords based Retrieval (AKR) Technique
Ranking problem
Composite Rank is a weighted mix of Coverage,
Citation Count and Reference Count
15

16.
Rec4LRW System Design - II
Task 2 - Finding Similar Papers based on Set of Papers Extended paper
discovery
problem
Multiple input
papers
Integrated
Discovery of
Similar Papers
(IDSP) Technique
IDSP
Technique
Similar papers
16

17.
Rec4LRW System Design - II
Task 3 - Shortlisting Articles from RL for Inclusion in Manuscript Cluster detection
problem
Final list of
papers from LR
Citation Network
based Shortlisting
(CNS) TechniqueIDSP
Technique
Unique and
important papers
17

18.
Rec4LRW System Design - III
Task Screens
Task 1
Task 2
Information
cue labels
Seed
Basket (SB)
18

19.
Rec4LRW System Design - III
Task Screens
Task 2
Task 3
Shared
Co-relations
Reading List
(RL)
19

20.
Rec4LRW System Design - III
Task Screens
Task 3
• Front end: PHP, HTML, CSS, JavaScript
• Backend: MySQL
• Processing layer: JAVA
• Java libraries: Apache Lucence (for BM25), Apache Mahout (for IBCF), Jung (for community
detection algorithm)
Cluster
viewing
option
20

21.
Study II – Rec4LRW Evaluation
21

22.
Study II - Dataset
• XML files provided by ACM
• Papers published in the period 1951 to 2011
• Total of 103,739 articles and corresponding 2,320,345 references
• Data was cleaned and transformed in MySQL
• References were parsed using AnyStyle parser
• All the seven base features were precomputed before Study II
22

23.
Study II – Pre-study
Evaluated Techniques
Label Abbr. Technique Description
A AKRv1 Basic AKR technique with weights WCC = 0.25, WRC=0.25, WCO = 0.5
B AKRv2 Basic AKR technique with weights WCC = 0.1, WRC=0.1, WCO = 0.8
C HAKRv1 HITS enhanced AKR technique boosted with weights WCC = 0.25, WRC=0.25, WCO = 0.5
D HAKRv2 HITS enhanced AKR technique boosted with weights WCC = 0.1, WRC=0.1, WCO = 0.8
E CFHITS IBCF technique boosted with HITS
F CFPR IBCF technique boosted with PageRank
G PR PageRank technique
Experiment Setup
• A total of 186 author-specified keywords from the ACM DL dataset were identified as the seed research topic
• The experiment was performed in three sequential steps.
1. Top 200 papers were retrieved using the BM25 similarity algorithm
2. Top 20 papers were identified using the specific ranking schemes of the seven techniques
3. The evaluation metrics were measured for the seven techniques
Evaluation Approach
• Number of Recent (R1), Popular (R2), Survey (R3) and Diverse (R4) papers were enumerated for each of the
186 topics and seven techniques
• Ranks were assigned to the technique based on the highest counts in each recommendation list
• The RankAggreg library was used to perform Rank Aggregation
23

24.
Study II – Part I (Pre-study)
Results
Paper Type (Requirement)
Optimal Aggregated Ranks Min. Obj. Function
Score1 2 3 4 5 6 7
Recent Papers (R1) B A C D E F G 10.66
Popular Papers (R2) F E C D G A B 11.89
Literature Survey Papers (R3) C G D A E F B 13.38
Diverse Papers (R4) C D G A B F E 12.15
• The HITS enhanced version of the AKR technique HAKRv1 (C) was the best all-round performing technique
• The HAKRv1 technique was particularly good for retrieving literature survey papers and papers from different
sub-topics while the basic AKRv1 technique (A) was good for retrieving recent papers
• The baseline CFPR technique (F) remains the best technique for retrieving popular papers
• The advantage of using weights has been shown
• AKR technique’s scalability is highlighted
24

25.
Study II – User Study Evaluation Goals
1. Ascertain the agreement percentages of the evaluation measures for the three tasks
and the overall system and identify whether the values are above a preset threshold
criteria of 75%
2. Test the hypothesis that students benefit more from the recommendation
tasks/system in comparison to staff
3. Measure the correlation between the measures and build a regression model with
‘agreeability on a good list’ as the dependent variable
4. Track the change in user perceptions between the three tasks
5. Compare the pre-study and post-study variables for understanding whether the
target participants are benefitted from the tasks
6. Identify the top most preferred and critical aspects of the task recommendations and
the system using the subjective feedback of the participants
25

26.
Study II - Details
• Rec4LRW system was made available over the internet
• Participants were recruited with intent to get worldwide audience
• Only researchers with paper authoring experience were recruited through a
pre-screening survey
• 230 researchers participated in the pre-screening survey
• 149 participants were deemed eligible and invited for the study
• Participants provided with a user guide
• All the three tasks were required to be executed by the participants
• Evaluation questionnaires embedded in the screen of each task of Rec4LRW
system
26

27.
Study II – Participant Demographics
Stage N
Task 1 132
Task 2 121
Task 3 119
Demographic Variable N
Position
Student 62 (47%)
Staff 70 (53%)
Experience Level
Beginner 15 (11.4%)
Intermediate 61 (46.2%)
Advanced 34 (25.8%)
Expert 22 (16.7%)
Discipline N
Computer Science & Information Systems 51 (38.6%)
Library and Information Studies 30 (22.7%)
Electrical & Electronic Engineering 30 (22.7%)
Communication & Media Studies 8 (6.1%)
Mechanical, Aeronautical & Manufacturing Engineering 5 (3.8%)
Biological Sciences 2 (1.5%)
Statistics & Operational Research 1 (0.8%)
Education 1 (0.8%)
Politics & International Studies 1 (0.8%)
Economics & Econometrics 1 (0.8%)
Civil & Structural Engineering 1 (0.8%)
Psychology 1 (0.8%)
Country N
Singapore 107 (81.1%)
India 4 (3%)
Malaysia 3 (2.3%)
Sri Lanka 3 (2.3%)
Pakistan 3 (2.3%)
Indonesia 2 (1.5%)
Germany 2 (1.5%)
Australia 1 (0.8%)
Iran 1 (0.8%)
Thailand 1 (0.8%)
China 1 (0.8%)
USA 1 (0.8%)
Canada 1 (0.8%)
Sweden 1 (0.8%)
Slovenia 1 (0.8%) 27

28.
Study II – Task Evaluation Measures
Common Measures
• Relevance
• Usefulness
• Good_List
Tasks 1 and 2
• Good_Spread
• Diversity
• Interdisciplinarity
• Popularity
• Recency
• Good_Mix
• Familiarity
• Novelty
• Serendipity
• Expansion_Required
• User_Satisfaction
Task 2 specific
• Seedbasket_Similarity
• Shared_Corelations
• Seedbasket_Usefulness
Task 3 specific
• Importance
• Certainty
• Shortlisting_Feature
28
1) From the displayed information, what features did
you like the most?
2) Please provide your personal feedback about the
execution of this task

29.
Study II – System Evaluation Measures
Effort to use the System (EUS)
• Convenience
• Effort_Required
• Mouse_Clicks
• Little_Time
• Much_Time
Perceived Usefulness (PU)
• Productivity_Improvability
• Enhance_Effectiveness
• Ease_Job
• Work_Usefulness
Perceived System Effectiveness (PSE)
• Recommend
• Pleasant_Experience
• Useless
• Awareness
• Better_Choice
• Findability
• Accomplish_Tasks
• Performance_Improvability
29

30.
Study II – Analysis Procedures
Quantitative Data
• Agreement Percentage (AP) calculated by only considering responses of 4
(‘Agree’) and 5 (‘Strongly Agree’) in the 5-point Likert scale
• Independent samples t-test for hypothesis testing
• Spearman coefficient for correlation measurement
• MLR used for the predictive models
– Paired samples t-test for model validation
Qualitative Data
• Descriptive coding method was used to code the participant feedback
• Two coders performed the coding in a sequential manner
Preferred Aspects (κ) Critical Aspects (κ)
Task 1 0.918 0.727
Task 2 0.930 0.758
Task 3 0.877 0.902
30

31.
Study II – Results for Goals 1 & 2
31

32.
Study II – Results for Goals 3 and 4
Predictors for “Good_List”
Task Independent Variables
Task 1 Recency, Novelty, Serendipity, Usefulness, User_Satisfaction
Task 2 Seedbasket_Similarity, Usefulness
Task 3 Relevance, Usefulness, Certainty
Transition of User Perception from Task 1 to 2
32

33.
Study II – Results for Goal 5
0 1 3 4 00
6
5
10
21
9
18
22
40 1
11
18
10 1 2
5
6
0
5
10
15
20
25
Count
1
2
3
4
5
0 3 5
20
30 3
9
30
41 2 7
21
20 0 3 1 2
0
5
10
15
20
25
30
35
Count 1
2
3
4
5
0 1 3 2 30 2
8
15
40 4
7
24
6
0 1 5
16
31 1 2 5 1
0
5
10
15
20
25
30
Count
1
2
3
4
5
Task 1
Task 2
Task 3
Need_Assistance
(pre study)
Vs.
Good_List
(post study)
33
Never Rarely Sometimes Often Always
Never Rarely Sometimes Often Always
Never Rarely Sometimes Often Always

34.
Study II – Results for Goal 6
Top 5 Preferred Aspects
Rank Task 1 (N=109) Task 2 (N=100) Task 3 (N=91)
1 Information Cue Labels (41%)
Shared Co-citations & Co-references
(28%)
Shortlisting Feature &
Recommendation Quality (24%)
2 Rich Metadata (21%) Recommendation Quality (27%) Information Cue Labels (15%)
3 Diversity of Papers (13%) Information Cue Labels (16%) View Papers in Clusters (11%)
4 Recommendation Quality (9%) Seed Basket (14%) Rich Metadata (7%)
5 Recency of Papers (4%) Rich Metadata (9%) Ranking of Papers (3%)
Rank Task 1 (N=109) Task 2 (N=100) Task 3 (N=91)
1 Broad topics not suitable (20%) Quality can be improved (16%)
Rote selection of papers for task
execution (16%)
2 Limited dataset (7%) Limited dataset (12%) Limited dataset (5%)
3 Quality can be improved (6%)
Recommendation algorithm could
include more dimensions (7%)
Algorithm can be improved (5%)
4 Different algorithm required (5%) Speed can be improved (7%) Not sure of the usefulness (4%)
5 Free-text search required (4%)
Repeated recommendations from Task 1
(3%)
UI can be improved (3%)
Top 5 Critical Aspects
34

35.
Contributions and Implications
• The Rec4LRW system and its recommendations adequately
satisfy the most affected user group – Students
• Addresses the piecemeal scholarship on scientific paper
recommender systems (SPRS)
• Proposes bridge between task requirements and IR/RS
algorithms
• The threefold intervention framework helps in integrating
research ideas from UI, IR and RS research areas
35

36.
Limitations
• Recommendation techniques do not cater to disciplinary
differences (if any)
• Recommendations could be biased to certain requirements
of the three tasks
• Non-user personalized techniques (not a serious issue)
• Evaluation study conducted with a limited set of research
topics
36

37.
SPRRF - Scientific Paper Retrieval and
Recommender Framework (SPRRF)
Distinct User
Groups
Usefulness of
Information Cue
Labels
Forced
Serendipity vs.
Natural
Serendipity
Learning
Algorithms vs.
Fixed-Logic
Algorithms
Inclusion of
Control
Features in UI
Inclusion of
Bibliometric
Data
Diversification
of Corpus
• Seven themes identified using holistic coding method
• SPRRF conceptualized as a mental model based on
the themes
• The framework needs to be validated
37

38.
Future Work
• Validation of the proposed SPRRF framework
• Longitudinal user evaluation studies
• Improvements in recommendation techniques
– Inclusion of more metrics
– More weights for customization
– Citation motivations
– Usage of open web standards
38

39.
Publications
Journal Papers
1. Raamkumar, A. S., Foo, S., & Pang, N. (2016). Survey on inadequate and omitted citations in manuscripts: a precursory study in identification of
tasks for a literature review and manuscript writing assistive system. Information Research, 21(4).
2. Raamkumar, A. S., Foo, S., & Pang, N. (2017). Using author-specified keywords in building an initial reading list of research papers in scientific
paper retrieval and recommender systems. Information Processing & Management, 53(3), 577-594.
3. Sesagiri Raamkumar, A., Foo, S., Pang, N. (2017). Evaluating a threefold intervention framework for assisting researchers in literature review and
manuscript preparatory tasks. Journal of Documentation, 73(3), 555-580.
4. Sesagiri Raamkumar, A., Foo, S., Pang, N. (2017). User Evaluation of a Task for Shortlisting Papers from Researcher’s Reading List for Citing in
Manuscripts. Aslib Journal of Information Management, 69(6).
5. Sesagiri Raamkumar, A., Foo, S., Pang, N. (2017). Can I have more of these please? Assisting researchers in finding similar research papers from
a seed basket of papers. The Electronic Library. Manuscript recommended for publication.
Conference Papers
1. Sesagiri Raamkumar, A., Foo, S., & Pang, N. (2015). Rec4LRW-scientific paper recommender system for literature review and writing. Frontiers in
Artificial Intelligence and Applications (Vol. 275).
2. Raamkumar, A. S., Foo, S., & Pang, N. (2015). Comparison of techniques for measuring research coverage of scientific papers: A case study. In
Digital Information Management (ICDIM), 2015 Tenth International Conference on (pp. 132-137). IEEE.
3. Raamkumar, A. S., Foo, S., & Pang, N. (2015). More Than Just Black and White: A Case for Grey Literature References in Scientific Paper
Information Retrieval Systems. In International Conference on Asian Digital Libraries (pp. 252-257). Springer, Cham.
4. Sesagiri Raamkumar, A., Foo, S., & Pang, N. (2016,). Making Literature Review and Manuscript Writing Tasks Easier for Novice Researchers
through Rec4LRW System. In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries (pp. 229-230). ACM.
5. Sesagiri Raamkumar, A., Foo, S., & Pang, N. (2016). What papers should I cite from my reading list? User evaluation of a manuscript preparatory
assistive task. In Proceedings of the Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital
Libraries (BIRNDL2016) (pp. 51–62).
39

40.
THANK YOU
40