Slides from my talk to the CWI Amsterdam Information Access group, 2015-04-23

1. Synthesizing knowledge from
disagreement
Jodi Schneider
ERCIM Marie Curie Postdoctoral Fellow, INRIA
jschneider@pobox.com
2015-04-23
CWI
Amsterdam

2. Overview
o My Background & Research Themes
o Structuring Evidence in Wikipedia Discussions
o Supporting Systematic Review of Biomedical Evidence
2

3. Themes in My Research
o How do people collaborate to generate knowledge?
o What counts as evidence in a given community?
o How can structuring evidence help synthesize info?
3

4. What knowledge should be included
in Wikipedia?
Jodi Schneider, Krystian Samp, Alexandre Passant, and Stefan Decker. “Arguments about Deletion:
How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups”. In CSCW
2013.
Jodi Schneider and Krystian Samp. “Alternative Interfaces for Deletion Discussions in Wikipedia:
Some Proposals Using Decision Factors. [Demo]” In WikiSym2012.
Jodi Schneider, Alexandre Passant, and Stefan Decker. “Deletion Discussions in Wikipedia:
Decision Factors and Outcomes.” In WikiSym2012.
4

5. Important since Wikipedia is widely used.
5

6. Wikipedia deletes articles.
6

7. Wikipedia deletes articles.
7

8. Wikipedia deletes articles.
8

9. Wikipedia deletes articles.
9

10. Example Deletion Discussion
10

11. Problem: Long, No-consensus Discussions

12. Problem: Long, No-consensus Discussions

13. Problem: Newcomers are confused about
Wikipedia’s standards.
o “Why should a local cricket club not have it's own
page on this website? Obviously a valid club and
been established for a while. Nothing offensive or
false on the page. All need to do is put in Emsworth
Cricket Club into a search engine and information
comes up. Why just because it is a small team
and not major does it not deserve it's own page
on here?” (sic)
o “At the end of the day the club has history which
being 200 years is just as special as a article on a
breed of dog or something similar.”
o “really is worth a mention. Especially on a
website, where pointless people ... gets a
mention.” (sic)
13

14. Problem: Newcomers are confused about
Wikipedia’s standards.
o “Why should a local cricket club not have it's own
page on this website? Obviously a valid club and
been established for a while. Nothing offensive or
false on the page. All need to do is put in Emsworth
Cricket Club into a search engine and information
comes up. Why just because it is a small team
and not major does it not deserve it's own page
on here?” (sic)
o “At the end of the day the club has history which
being 200 years is just as special as a article on a
breed of dog or something similar.”
o “really is worth a mention. Especially on a
website, where pointless people ... gets a
mention.” (sic)
14

15. Problem: Newcomers are confused about
Wikipedia’s standards.
15

16. Problem: Newcomers are confused about
Wikipedia’s standards.
16

17. Problem: Newcomers are confused about
Wikipedia’s standards.
17

18. Problem Summary
o Long, no-consensus discussions
 Summarize discussions
o Newcomers are confused about Wikipedia's standards
 Make article criteria more explicit
18

19. Approach: Structure Evidence
1. Understand what evidence the community uses to
establish knowledge.
2. Structure the evidence.
3. Build a computer support system.
4. Test and refine the system.
19

20. Approach: Structure Evidence
1. Understand what evidence the community uses
to establish knowledge.
2. Structure the evidence.
3. Build a computer support system.
4. Test and refine the system.
20

21. Sample Corpus
o 72 discussions started on 1 day.
Each discussion has
• 3–33 messages
• 2–15 participants
o In total, 741 messages contributed by 244 users.
Each message has
• 3–350+ words
o 98 printed A4 sheets
21

22. Structuring the Data: Annotation
o Content analysis of the corpus
o Compare two different annotation approaches
o Iterative annotation
• Multiple annotators
• Refine to get good inter-annotator agreement
• 4 rounds of annotation
22

23. 2 Types of Annotation
o 1. Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
• Informal argumentation
(philosophical & computational argumentation)
• Identify & prevent errors in reasoning (fallacies)
• 60 patterns
o 2. Factors Analysis
(Ashley 1991)
• Case-based reasoning
• E.g. factors for deciding cases in trade secret law,
favoring either party (the plaintiff or the defendant).
23

24. 2 Types of Annotation
1. Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
Informal argumentation
(philosophical & computational argumentation)
Identify & prevent errors in reasoning (fallacies)
60 patterns
o 2. Factors Analysis
(Ashley 1991)
• Case-based reasoning
• E.g. factors for deciding cases in trade secret law,
favoring either party (the plaintiff or the defendant).
24

25. Annotation: Factors

26. Factor Example (used to justify ‘keep’)
Notability Anyone covered by another
encyclopedic reference is considered
notable enough for inclusion in
Wikipedia.
Sources Basic information about this album at a
minimum is certainly verifiable, it's a
major label release, and a highly
notable band.
Maintenance …this article is savable but at its
current state, needs a lot of
improvement.
Bias It is by no means spam (it does not
promote the products).
Other I'm advocating a blanket “hangon” for
all articles on newly-drafted players…
Jodi Schneider, Alexandre Passant & Stefan Decker
Deletion Discussions in Wikipedia: Decision Factors and Outcomes
4 Key Factors (& “Other”)
26

27. Decision factors articulate values/criteria.
o 4 Factors in Deletion Discussions cover:
• 91% of comments
• 70% of discussions
o Readers who understand these criteria:
• Understand what content is appropriate.
• Are less likely to have content deleted, and less likely to
take deletion personally.
27

28. To structure the data, we chose factors.
o 1. Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
• Most appropriate for writing support
• 15 categories + 2 non-argumentative categories
• Detailed analysis of content
o 2. Factors Analysis
o (drawing on Ashley 1991)
• Close to the community rules & policies
• 4 categories + 1 catchall
• Good domain coverage
28

29. Approach: Structure Evidence
1. Understand what evidence the community uses
to establish knowledge.
2. Structure the evidence.
3. Build a computer support system.
4. Test and refine the system.
29

30. Approach: Structure Evidence
1. Understand what evidence the community uses to
establish knowledge.
2. Structure the evidence.
3. Build a computer support system.
4. Test and refine the system.
30

31. Add a discussion summary.
31

32. Data Model
32

33. Approach: Structure Evidence
1. Understand what evidence the community uses to
establish knowledge.
2. Structure the evidence.
3. Build a computer support system.
4. Test and refine the system.
33

34. Approach: Structure Evidence
1. Understand what evidence the community uses to
establish knowledge.
2. Structure the evidence.
3. Build a computer support system.
4. Test and refine the system.
34

35. Build a computer support system.
Original
Discussion
Ontology
Semantic
Enrichment
Semantically
Enriched
RDFa
Querying
Queryable
User Interface
With Barchart

36. We add a discussion summary…
36

37. …by annotating this original content….
37

38. …to semantically enrich messages.
38

39. …to semantically enrich messages.
39

40. …to semantically enrich messages.
40

41. …to semantically enrich messages.
41

42. Our discussion summary…
42

43. … gives more detail for each decision factor.
On click, open the comments
with that decision factor.

44. Count & list by decision factor using
JavaScript queries
44

45. Query to generate the summary.
45

46. Query to generate the summary.
46

47. Approach: Structure Evidence
1. Understand what evidence the community uses to
establish knowledge.
2. Structure the evidence.
3. Build a computer support system.
4. Test and refine the system.
47

48. Test our Experimental System…

49. against this Control System.

50. Experimental Design
50

51. Experimental design
51
{

52. 52

53. PU* - Perceived usefulness
PE* - Perceived ease of use
DC -Decision completeness
PF - Perceived effort
IC* - Information
completeness
Statistical Significance
PU* p < .001
PE* p .001
IC* p .039
53

54. Final Survey
54

55. Results: 84% prefer our system.
“Information is structured and I can quickly get an
overview of the key arguments.”
“The ability to navigate the comments made it a bit
easier to filter my mind set and to come to a
conclusion.”
“It offers the structure needed to consider each factor
separately, thus making the decision easier. Also, the
number of comments per factor offers a quick
indication of the relevance and the deepness of the
decision.”
16/19, based on a 20 participant user test.
1 participant did not take the final survey
55

56. Approach: Structure Evidence
1. Understand what evidence the community uses to
establish knowledge.
2. Structure the evidence.
3. Build a computer support system.
4. Test…
… & refine the system.
56

57. Summary
o Information technology can organize information
based on a community’s key decision factors.
o In Wikipedia, we developed an alternate interface for
deletion discussions.
o In Wikipedia, 4 questions are used to evaluate
borderline articles:
o Notability – Is the topic appropriate for our encyclopedia?
o Sources – Is the article well-sourced?
o Maintenance – Can we maintain this article?
o Bias – Is the article neutral? POV appropriately weighted?
57

58. Summary: Our Process
1. Get to know a community and its needs.
Ethnography
1. Structure the data.
Annotation & ontology development
1. Build a computer support system.
Web standards:
HTML, JavaScript, RDF/OWL, SPARQL
1. Test & refine the system.
Human computer interaction
58

59. SUPPORTING SYSTEMATIC
REVIEW OF BIOMEDICAL
EVIDENCE
59

60. 2000+ new papers each day
60

61. Info overload now goes beyond
papers
Bastian, Glasziou, and Chalmers. "75 trials and 11 systematic reviews
a day: how will we ever keep up?." PLoS medicine 7.9 (2010): e1000326.

62. For medication safety, how to
structure evidence on drug-drug
interactions and keep it up-to-date?
Jodi Schneider, Paolo Ciccarese, Tim Clark and Richard D. Boyce. “Using the
Micropublications ontology and the Open Annotation Data Model to represent evidence
within a drug-drug interaction knowledge base.” 4th Workshop on Linked Science 2014—
Making Sense Out of Data (LISC2014) at ISWC 2014
Mathias Brochhausen, Jodi Schneider, Daniel Malone, Philip E. Empey, William R. Hogan
and Richard D. Boyce “Towards a foundational representation of potential drug-drug
interaction knowledge.” First International Workshop on Drug Interaction Knowledge
Representation (DIKR-2014) at the International Conference on Biomedical Ontologies
(ICBO 2014)
Jodi Schneider, Carol Collins, Lisa Hines, John R Horn and Richard Boyce. “Modeling
Arguments in Scientific Papers to Support Pharmacists.” at ArgDiaP 2014, The 12th
ArgDiaP Conference: From Real Data to Argument Mining, Warsaw, Poland
62

63. Part of a Larger Effort
o “Addressing gaps in clinically useful evidence on
drug-drug interactions”
o 4-year project, U.S. National Library of Medicine R01
grant
(PI, Richard Boyce; 1R01LM011838-01)
o Since February 2013:
evidence panel of domain experts
(Carol Collins, Lisa Hines, John R Horn, Phil Empey)
& informaticists
(Tim Clark, Paolo Ciccarese, Jodi Schneider)
o Programmer: Yifan Ning

64. Prescribers check for known drug interactions.
64

65. Prescribers consult drug interaction references
which are maintained by expert pharmacists.
Medscape EpocratesMicromedex 2.0
65

66. Prescribers consult drug interaction references
which are maintained by expert pharmacists.
Medscape EpocratesMicromedex 2.0
66

67. Goals
o Support evidence-based updates to
drug-interaction reference databases.
o Make sense of the EVIDENCE:
• New clinical trials
• Adverse drug event reports
• Drug product labels
• FDA regulatory updates
http://jama.jamanetwork.com/article.aspx?articleid=18345467

68. o Evidence
68

69. Evidence Base Competency Questions
o 40 competency questions, such as:
• List all evidence by drug, drug pair, …
• List all default assumptions
(assertions not supported by evidence)
• Which single evidence items act as as support or rebuttal
for multiple assertions of type X?
(e.g., substrate_of assertions)
• What data, methods, materials, were used in the study
reported in evidence item X?
• Which research group conducted the study reported in
evidence item X?
• Show me what evidence has been deprecated since my
last visit?
• Which assertions are supported by a specific FDA
guidance statement?
69

70. An Ontology for Representing Evidence
Clark, Ciccarese, Goble (2014) Micropublications: a semantic model for claims, evidence, arguments and
annotations in biomedical communications
70

71. An Ontology for Representing Evidence
71
Clark, Ciccarese, Goble (2014) Micropublications: a semantic model for claims, evidence, arguments and
annotations in biomedical communications

72. 72

73. 73

74. 74

75. 75

76. 76

77. 77

78. 78

79. 79

80. 80

81. o Evidence
81
7.19 Drugs Metabolized by Cytochrome P4502D6
In vitro studies did not reveal an inhibitory effect of
escitalopram on CYP2D6.

82. 82

83. Next steps
o Continuing data model development & testing.
o NLP support: Create a pipeline for extracting
potential drug-drug interaction mentions from
scientific & clinical literature.
o NLP + "expertsourcing" and crowdsourcing
(distributed annotation).
o Test annotation tools: usability for domain experts.
o Resolving links to paywalled PDFs.
83

85. Open Annotation Data Model
http://www.openannotation.org/spec/core/

86. Annotation: Argumentation Schemes

87. “Rule” Argumentation Scheme

88. “Evidence” Argumentation Scheme

89. Evidence + Rule -> Conclusion

90. Walton’s Argumentation Schemes
Example Argumentation Scheme:
Argument from Rules – “we apply rule X”
Critical Questions
1. Does the rule require carrying out this type of
action?
2. Are there other established rules that might conflict
with or override this one?
3. Are there extenuating circumstances or an excuse
for noncompliance?
Walton, Reed, and Macagno 2008

91. Walton’s Argumentation Schemes
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker.
“Arguments about Deletion: How Experience Improves the Acceptability of Arguments in
Ad-hoc Online Task Groups”. In CSCW 2013.
! "#$%&' ()*+((&"' *"&, +-&' . &
! "#$%&' ()*"+%), -./&' 0&)(+)123+(4&5.5 67897:
! "#$%&' ()*"+%); $<&5 6=87>:
? +(& 6@8=7:
! "#$%&' (A(.+' )*"+%)BA<$&5 C89>:
! "#$%&' ()*"+%)? &&/)*+")1&<3 C869:
! "#$%&' ()*"+%)D.A5 @8EF:
? +)"&A5+' )#.-&' @899:
! "#$%&' ()*"+%)G+5.(.+' )(+)H' +I @8>J :
! "#$%&' ()*"+%)G"&0&/&' ( @8>J :
! "#$%&' ()*"+%)K#' +"A' 0& 987F:
! "#$%&' ()*"+%)L+%3+5.(.+' 98J =:
! "#$%&' ()*"+%)LA$5&)(+), **&0( 98@6:
! "#$%&' ()*"+%)! ' A<+#2 989@:
! "#$%&' ()*"+%)MA5(& 989@:
G"A0(.0A<); &A5+' .' # 989@:
! "#8)*"+%)B&"NA<)L<A55.*.0A(.+' 98>=: