Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

A task-based scientific paper recommender system for literature review and manuscript preparation

216 visualizaciones

Publicado el

My PhD oral defense presentation (as of Oct 3rd 2017)

The dissertation can be requested at this link https://www.researchgate.net/publication/323308750_A_task-based_scientific_paper_recommender_system_for_literature_review_and_manuscript_preparation

Publicado en: Tecnología
  • Sé el primero en comentar

A task-based scientific paper recommender system for literature review and manuscript preparation

  1. 1. A Task-based Scientific Paper Recommender System for Literature Review and Manuscript Preparation Aravind SESAGIRI RAAMKUMAR PhD Candidate Oral Examination Presentation for fulfillment of PhD October 3rd 2017
  2. 2. Citation recommendations Research paper recommender systems Recommending papers {Ad-hoc search} Background “How to get the best set of relevant documents for a researcher’s literature review and publication purposes?” SPRS Research Area More than 250 papers Process-based Interventions Technology-oriented Interventions 2
  3. 3. Identify research opportunities Find collaborators Secure support Review the literature Collect research data Analyze research data Disseminate findings Manage the research process Related Work Classification (1/2) Active and Explicit Information Needs (AEIN) in the Research Lifecycle A) Active and Explicit Information Needs (AEIN) B) Passive and Implicit Information Needs (PIIN) <Recommendation of Scholarly Objects> • Building a reading list for literature review • Finding similar papers : : : • Searching papers based on input text • Publication Venues • Citation Context 3
  4. 4. Related Work Classification (2/2) Passive and Implicit Information Needs (PIIN) • User Footprint • Researcher’s Publication History • Social Network of Authors • Social Tags • Reference Management Systems 4
  5. 5. Research Gaps in SPRS Studies Consolidated Framework for Contextual Dimensions Lack of Connectivity between Tasks Lack of Relation(s) between Tasks and RS Filtering Mechanisms Absence of Article Type as an Input Dimension 5
  6. 6. Research Objectives • To identify an appropriate method to map the identified LR and MP tasks to relevant IR/RS algorithms – RQ1: What are the key search tasks of researchers in the literature review and publication lifecycle? – RQ2: How to relate the identified tasks of researchers to IR/RS algorithms? • To evaluate whether the performance of the proposed recommendation techniques for the tasks and the overall system were at the expected level – RQ3: Do the proposed recommendation techniques of the relevant tasks outperform the existing baseline approaches in system-based evaluation? – RQ4: Do the proposed recommendation techniques and the overall system meet the expected standards in user- based evaluation? Study I Study II Rec4LRW System 6
  7. 7. Study I - Survey on Inadequate and Omitted Citations (IOC) in Manuscripts Authors Reviewers Problems in research quality Manuscripts with improper LR 7
  8. 8. Study I - Survey on Inadequate and Omitted Citations (IOC) in Manuscripts Aims • What are the critical instances of IOC? • Do the critical instances and reasons of IOC in research manuscripts relate with the scenarios/tasks where researchers need external assistance in finding papers? • Identify the prominent information sources • What is the researchers’ awareness level of available recommendation services for research papers? 8
  9. 9. Study I Details • Single center data collection conducted for two months • Only researchers with paper authoring experience were recruited • 207 NTU researchers participated in the study 71% of the participants answered from both reviewer and author perspectives • Survey questionnaire comprising of 31 questions • Agreeability measured in 5-point Likert scale • Data analyses through one-sample t-test with test value of either 2 or 3 9
  10. 10. Study I - Results Instances of IOC • Authors viewpoint – Missed citing seminal and topically-similar papers in journal manuscripts • Reviewers viewpoint – Missed citing seminal, topically-similar papers in all manuscripts – Insufficient and irrelevant papers in the LR of all manuscripts Effects of IOC • Reviewers viewpoint – Manuscripts are sent back for revision due to missing citations 10
  11. 11. Study I - Results Need for External Assistance in Finding Papers • Authors required support for the below papers:- 1. Interdisciplinary papers 2. Topically-similar papers 3. Seminal papers 4. Citations for placeholders in manuscripts 5. Necessary citations meant for inclusion in manuscripts Usage of Academic Information Sources • Researchers used the below sources in the order of usage 1. Google Scholar 2. ScienceDirect 3. Web of Science 4. SpringerLink • 62% of the participants have never used SPRS services 11
  12. 12. Study I – Key Findings • Researchers need help in finding interdisciplinary, topically- similar and seminal papers • Generating reading list (seminal papers) and finding similar papers are two necessary LR search tasks for the proposed system • Shortlisting papers from final reading list for inclusion in manuscript, selected as third task for the proposed system • Google Scholar’s simplistic UI makes it the most used information source and ideal choice for UI design of a new assistive system 12
  13. 13. Rec4LRW Design and Development Task Redesign Task Interconnectivity Informational Display Features 13
  14. 14. Rec4LRW System Design - I Base Features • Plug and Play concept • Features represent different characteristic of paper and its relations to references and citations • Grey Literature Percentage, Coverage, Textual Similarity and Specificity are novel features • New features can be added as required 14
  15. 15. Rec4LRW System Design - II Task 1 -Building an Initial Reading List of Research Papers Popular papers Recent papers Survey papers Diverse papers Use of Okapi BM25 Similarity Score to retrieve top 200 matching papers Requirements Author-specified Keywords based Retrieval (AKR) Technique Ranking problem Composite Rank is a weighted mix of Coverage, Citation Count and Reference Count 15
  16. 16. Rec4LRW System Design - II Task 2 - Finding Similar Papers based on Set of Papers Extended paper discovery problem Multiple input papers Integrated Discovery of Similar Papers (IDSP) Technique IDSP Technique Similar papers 16
  17. 17. Rec4LRW System Design - II Task 3 - Shortlisting Articles from RL for Inclusion in Manuscript Cluster detection problem Final list of papers from LR Citation Network based Shortlisting (CNS) TechniqueIDSP Technique Unique and important papers 17
  18. 18. Rec4LRW System Design - III Task Screens Task 1 Task 2 Information cue labels Seed Basket (SB) 18
  19. 19. Rec4LRW System Design - III Task Screens Task 2 Task 3 Shared Co-relations Reading List (RL) 19
  20. 20. Rec4LRW System Design - III Task Screens Task 3 • Front end: PHP, HTML, CSS, JavaScript • Backend: MySQL • Processing layer: JAVA • Java libraries: Apache Lucence (for BM25), Apache Mahout (for IBCF), Jung (for community detection algorithm) Cluster viewing option 20
  21. 21. Study II – Rec4LRW Evaluation 21
  22. 22. Study II - Dataset • XML files provided by ACM • Papers published in the period 1951 to 2011 • Total of 103,739 articles and corresponding 2,320,345 references • Data was cleaned and transformed in MySQL • References were parsed using AnyStyle parser • All the seven base features were precomputed before Study II 22
  23. 23. Study II – Pre-study Evaluated Techniques Label Abbr. Technique Description A AKRv1 Basic AKR technique with weights WCC = 0.25, WRC=0.25, WCO = 0.5 B AKRv2 Basic AKR technique with weights WCC = 0.1, WRC=0.1, WCO = 0.8 C HAKRv1 HITS enhanced AKR technique boosted with weights WCC = 0.25, WRC=0.25, WCO = 0.5 D HAKRv2 HITS enhanced AKR technique boosted with weights WCC = 0.1, WRC=0.1, WCO = 0.8 E CFHITS IBCF technique boosted with HITS F CFPR IBCF technique boosted with PageRank G PR PageRank technique Experiment Setup • A total of 186 author-specified keywords from the ACM DL dataset were identified as the seed research topic • The experiment was performed in three sequential steps. 1. Top 200 papers were retrieved using the BM25 similarity algorithm 2. Top 20 papers were identified using the specific ranking schemes of the seven techniques 3. The evaluation metrics were measured for the seven techniques Evaluation Approach • Number of Recent (R1), Popular (R2), Survey (R3) and Diverse (R4) papers were enumerated for each of the 186 topics and seven techniques • Ranks were assigned to the technique based on the highest counts in each recommendation list • The RankAggreg library was used to perform Rank Aggregation 23
  24. 24. Study II – Part I (Pre-study) Results Paper Type (Requirement) Optimal Aggregated Ranks Min. Obj. Function Score1 2 3 4 5 6 7 Recent Papers (R1) B A C D E F G 10.66 Popular Papers (R2) F E C D G A B 11.89 Literature Survey Papers (R3) C G D A E F B 13.38 Diverse Papers (R4) C D G A B F E 12.15 • The HITS enhanced version of the AKR technique HAKRv1 (C) was the best all-round performing technique • The HAKRv1 technique was particularly good for retrieving literature survey papers and papers from different sub-topics while the basic AKRv1 technique (A) was good for retrieving recent papers • The baseline CFPR technique (F) remains the best technique for retrieving popular papers • The advantage of using weights has been shown • AKR technique’s scalability is highlighted 24
  25. 25. Study II – User Study Evaluation Goals 1. Ascertain the agreement percentages of the evaluation measures for the three tasks and the overall system and identify whether the values are above a preset threshold criteria of 75% 2. Test the hypothesis that students benefit more from the recommendation tasks/system in comparison to staff 3. Measure the correlation between the measures and build a regression model with ‘agreeability on a good list’ as the dependent variable 4. Track the change in user perceptions between the three tasks 5. Compare the pre-study and post-study variables for understanding whether the target participants are benefitted from the tasks 6. Identify the top most preferred and critical aspects of the task recommendations and the system using the subjective feedback of the participants 25
  26. 26. Study II - Details • Rec4LRW system was made available over the internet • Participants were recruited with intent to get worldwide audience • Only researchers with paper authoring experience were recruited through a pre-screening survey • 230 researchers participated in the pre-screening survey • 149 participants were deemed eligible and invited for the study • Participants provided with a user guide • All the three tasks were required to be executed by the participants • Evaluation questionnaires embedded in the screen of each task of Rec4LRW system 26
  27. 27. Study II – Participant Demographics Stage N Task 1 132 Task 2 121 Task 3 119 Demographic Variable N Position Student 62 (47%) Staff 70 (53%) Experience Level Beginner 15 (11.4%) Intermediate 61 (46.2%) Advanced 34 (25.8%) Expert 22 (16.7%) Discipline N Computer Science & Information Systems 51 (38.6%) Library and Information Studies 30 (22.7%) Electrical & Electronic Engineering 30 (22.7%) Communication & Media Studies 8 (6.1%) Mechanical, Aeronautical & Manufacturing Engineering 5 (3.8%) Biological Sciences 2 (1.5%) Statistics & Operational Research 1 (0.8%) Education 1 (0.8%) Politics & International Studies 1 (0.8%) Economics & Econometrics 1 (0.8%) Civil & Structural Engineering 1 (0.8%) Psychology 1 (0.8%) Country N Singapore 107 (81.1%) India 4 (3%) Malaysia 3 (2.3%) Sri Lanka 3 (2.3%) Pakistan 3 (2.3%) Indonesia 2 (1.5%) Germany 2 (1.5%) Australia 1 (0.8%) Iran 1 (0.8%) Thailand 1 (0.8%) China 1 (0.8%) USA 1 (0.8%) Canada 1 (0.8%) Sweden 1 (0.8%) Slovenia 1 (0.8%) 27
  28. 28. Study II – Task Evaluation Measures Common Measures • Relevance • Usefulness • Good_List Tasks 1 and 2 • Good_Spread • Diversity • Interdisciplinarity • Popularity • Recency • Good_Mix • Familiarity • Novelty • Serendipity • Expansion_Required • User_Satisfaction Task 2 specific • Seedbasket_Similarity • Shared_Corelations • Seedbasket_Usefulness Task 3 specific • Importance • Certainty • Shortlisting_Feature 28 1) From the displayed information, what features did you like the most? 2) Please provide your personal feedback about the execution of this task
  29. 29. Study II – System Evaluation Measures Effort to use the System (EUS) • Convenience • Effort_Required • Mouse_Clicks • Little_Time • Much_Time Perceived Usefulness (PU) • Productivity_Improvability • Enhance_Effectiveness • Ease_Job • Work_Usefulness Perceived System Effectiveness (PSE) • Recommend • Pleasant_Experience • Useless • Awareness • Better_Choice • Findability • Accomplish_Tasks • Performance_Improvability 29
  30. 30. Study II – Analysis Procedures Quantitative Data • Agreement Percentage (AP) calculated by only considering responses of 4 (‘Agree’) and 5 (‘Strongly Agree’) in the 5-point Likert scale • Independent samples t-test for hypothesis testing • Spearman coefficient for correlation measurement • MLR used for the predictive models – Paired samples t-test for model validation Qualitative Data • Descriptive coding method was used to code the participant feedback • Two coders performed the coding in a sequential manner Preferred Aspects (κ) Critical Aspects (κ) Task 1 0.918 0.727 Task 2 0.930 0.758 Task 3 0.877 0.902 30
  31. 31. Study II – Results for Goals 1 & 2 31
  32. 32. Study II – Results for Goals 3 and 4 Predictors for “Good_List” Task Independent Variables Task 1 Recency, Novelty, Serendipity, Usefulness, User_Satisfaction Task 2 Seedbasket_Similarity, Usefulness Task 3 Relevance, Usefulness, Certainty Transition of User Perception from Task 1 to 2 32
  33. 33. Study II – Results for Goal 5 0 1 3 4 00 6 5 10 21 9 18 22 40 1 11 18 10 1 2 5 6 0 5 10 15 20 25 Count 1 2 3 4 5 0 3 5 20 30 3 9 30 41 2 7 21 20 0 3 1 2 0 5 10 15 20 25 30 35 Count 1 2 3 4 5 0 1 3 2 30 2 8 15 40 4 7 24 6 0 1 5 16 31 1 2 5 1 0 5 10 15 20 25 30 Count 1 2 3 4 5 Task 1 Task 2 Task 3 Need_Assistance (pre study) Vs. Good_List (post study) 33 Never Rarely Sometimes Often Always Never Rarely Sometimes Often Always Never Rarely Sometimes Often Always
  34. 34. Study II – Results for Goal 6 Top 5 Preferred Aspects Rank Task 1 (N=109) Task 2 (N=100) Task 3 (N=91) 1 Information Cue Labels (41%) Shared Co-citations & Co-references (28%) Shortlisting Feature & Recommendation Quality (24%) 2 Rich Metadata (21%) Recommendation Quality (27%) Information Cue Labels (15%) 3 Diversity of Papers (13%) Information Cue Labels (16%) View Papers in Clusters (11%) 4 Recommendation Quality (9%) Seed Basket (14%) Rich Metadata (7%) 5 Recency of Papers (4%) Rich Metadata (9%) Ranking of Papers (3%) Rank Task 1 (N=109) Task 2 (N=100) Task 3 (N=91) 1 Broad topics not suitable (20%) Quality can be improved (16%) Rote selection of papers for task execution (16%) 2 Limited dataset (7%) Limited dataset (12%) Limited dataset (5%) 3 Quality can be improved (6%) Recommendation algorithm could include more dimensions (7%) Algorithm can be improved (5%) 4 Different algorithm required (5%) Speed can be improved (7%) Not sure of the usefulness (4%) 5 Free-text search required (4%) Repeated recommendations from Task 1 (3%) UI can be improved (3%) Top 5 Critical Aspects 34
  35. 35. Contributions and Implications • The Rec4LRW system and its recommendations adequately satisfy the most affected user group – Students • Addresses the piecemeal scholarship on scientific paper recommender systems (SPRS) • Proposes bridge between task requirements and IR/RS algorithms • The threefold intervention framework helps in integrating research ideas from UI, IR and RS research areas 35
  36. 36. Limitations • Recommendation techniques do not cater to disciplinary differences (if any) • Recommendations could be biased to certain requirements of the three tasks • Non-user personalized techniques (not a serious issue) • Evaluation study conducted with a limited set of research topics 36
  37. 37. SPRRF - Scientific Paper Retrieval and Recommender Framework (SPRRF) Distinct User Groups Usefulness of Information Cue Labels Forced Serendipity vs. Natural Serendipity Learning Algorithms vs. Fixed-Logic Algorithms Inclusion of Control Features in UI Inclusion of Bibliometric Data Diversification of Corpus • Seven themes identified using holistic coding method • SPRRF conceptualized as a mental model based on the themes • The framework needs to be validated 37
  38. 38. Future Work • Validation of the proposed SPRRF framework • Longitudinal user evaluation studies • Improvements in recommendation techniques – Inclusion of more metrics – More weights for customization – Citation motivations – Usage of open web standards 38
  39. 39. Publications Journal Papers 1. Raamkumar, A. S., Foo, S., & Pang, N. (2016). Survey on inadequate and omitted citations in manuscripts: a precursory study in identification of tasks for a literature review and manuscript writing assistive system. Information Research, 21(4). 2. Raamkumar, A. S., Foo, S., & Pang, N. (2017). Using author-specified keywords in building an initial reading list of research papers in scientific paper retrieval and recommender systems. Information Processing & Management, 53(3), 577-594. 3. Sesagiri Raamkumar, A., Foo, S., Pang, N. (2017). Evaluating a threefold intervention framework for assisting researchers in literature review and manuscript preparatory tasks. Journal of Documentation, 73(3), 555-580. 4. Sesagiri Raamkumar, A., Foo, S., Pang, N. (2017). User Evaluation of a Task for Shortlisting Papers from Researcher’s Reading List for Citing in Manuscripts. Aslib Journal of Information Management, 69(6). 5. Sesagiri Raamkumar, A., Foo, S., Pang, N. (2017). Can I have more of these please? Assisting researchers in finding similar research papers from a seed basket of papers. The Electronic Library. Manuscript recommended for publication. Conference Papers 1. Sesagiri Raamkumar, A., Foo, S., & Pang, N. (2015). Rec4LRW-scientific paper recommender system for literature review and writing. Frontiers in Artificial Intelligence and Applications (Vol. 275). 2. Raamkumar, A. S., Foo, S., & Pang, N. (2015). Comparison of techniques for measuring research coverage of scientific papers: A case study. In Digital Information Management (ICDIM), 2015 Tenth International Conference on (pp. 132-137). IEEE. 3. Raamkumar, A. S., Foo, S., & Pang, N. (2015). More Than Just Black and White: A Case for Grey Literature References in Scientific Paper Information Retrieval Systems. In International Conference on Asian Digital Libraries (pp. 252-257). Springer, Cham. 4. Sesagiri Raamkumar, A., Foo, S., & Pang, N. (2016,). Making Literature Review and Manuscript Writing Tasks Easier for Novice Researchers through Rec4LRW System. In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries (pp. 229-230). ACM. 5. Sesagiri Raamkumar, A., Foo, S., & Pang, N. (2016). What papers should I cite from my reading list? User evaluation of a manuscript preparatory assistive task. In Proceedings of the Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL2016) (pp. 51–62). 39
  40. 40. THANK YOU 40

×