2. What I’m looking for
Digital Enterprise Research Institute www.deri.ie
Scoping & focus
Detailed mentoring on CSCW/HCC methodologies
Interviewing
Qualitative Research
Statistics
Suggestions for evaluating my work
2
4. Should we delete this article?
Digital Enterprise Research Institute www.deri.ie
4
5. Improving deletion discussions
Digital Enterprise Research Institute www.deri.ie
Main problems:
Newcomers who don’t know how to argue
Overwhelm of long discussions
Discussions that happen over and over again
Deletion as quality control
Large number of discussions - ~500/week
5
6. Deletion argument
Digital Enterprise Research Institute www.deri.ie
[Delete the article]...hasn't
played since 2008. His
66-73 record is far from
stellar and, in my opinion,
does not merit an article.
>>He pitched last month and
plays for the Venezuelan
League. This meets our
article criteria.
6
7. Goals
Digital Enterprise Research Institute www.deri.ie
Newcomers who don’t know how to argue
Characterize the “good” and “bad” arguments
Develop argument templates
Provide guidance and support for new users in properly
structuring arguments according to Wikipedia’s rhetorical
standards
Overwhelm of long discussions
Develop a claims/argument explorer
Discussions that happen over and over again
Prototype an argument bot
Populate argument maps with mixed-initiative claims
extraction
7
8. Overview
Digital Enterprise Research Institute www.deri.ie
Corpus:
All Wikipedia deletion discussions from January 29, 2011
Perspectives/approaches:
Argumentation
CSCW/HCC
Text analytics
Ontologies/Social Semantic Web
8
9. Current work
Digital Enterprise Research Institute www.deri.ie
Analysis of the corpus
Argument schemes (e.g. expert opinion)
Factors (e.g. notability, uniqueness)
Newcomer’s arguments
Interviews
Administrators
Experienced users
Argument exploration
Text mining cue words (‘however’, ‘therefore’,…)
Architecture
Ontology development
“Standpoints Web”
9
10. Standpoint
Digital Enterprise Research Institute www.deri.ie
[Delete the article]...hasn't
played since 2008. His
66-73 record is far from
stellar and, in my opinion,
does not merit an article.
Proposition: does not merit
an article
Justification: hasn’t played
since 2008, bad record
10
11. Opposing standpoint
Digital Enterprise Research Institute www.deri.ie
>>He pitched last month and
plays for the Venezuelan
League. This meets our
article criteria.
Proposition: keep the article
Justification: meets our article
criteria
11
12.
13. Possible applications
Digital Enterprise Research Institute www.deri.ie
Visualize decision-making
Highlight controversies
Query opinions and arguments
Discuss arguments interactively with a bot
Calculate the “best” options
Analyze, extract, and represent disagreement
13
15. Acknowledgments
Digital Enterprise Research Institute www.deri.ie
Thanks to our collaborators!
Katie Atkinson, Trevor Bench-Capon, Adam Wyner (Liverpool)
DERI Social Software Unit
Rhetorical Structure, W3C Health Care and Life Sciences
Funding
Science Foundation Ireland Grant No. SFI/08/CE/I1380
(Líon-2)
Short-term scientific mission (STSM 1868) from the COST
Action ICO801 on Agreement Technologies
15
17. “ELIZA for arguments”
Digital Enterprise Research Institute www.deri.ie
Snaith, Lawrence, & Reed, “Mixed initiative argument in public deliberation,” ODET 2010
17
18. Highlight Controversies
Digital Enterprise Research Institute www.deri.ie
Ennals, R., Trushkowsky, B., & Agosta, J. M. (2010). Highlighting Disputed
Claims on the Web. In WICOW at WWW 2010.
19. Transform Debates into
Argument Frameworks
Digital Enterprise Research Institute www.deri.ie
(1) Households should pay tax for
their garbage.
(4) (1)
Paying tax for garbage increases
recycling, so households should Arrow: premise
pay.
(3) (1)
Wyner, van Engers, & Bahreini.
Recycling more is good, so people From Policy-making Statements
to First-order Logic.
should pay tax for their garbage. EGOV 2010
19
20. Calculate best options
(non-contradictory opinions)
Digital Enterprise Research Institute www.deri.ie
Wyner, van Engers, & Bahreini.
From Policy-making Statements to First-order Logic.
EGOV 2010
20
21. Claims Extraction
Digital Enterprise Research Institute www.deri.ie
Cue words (“Hence Jaffa Cakes are cakes.”) [Marcu]
Rhetorical Structure Theory
25. Case Study
Digital Enterprise Research Institute www.deri.ie
Understand:
interviews, observation, and content analysis
Intervene:
Implement & test the Standpoints Web architecture
on Wikipedia deletion discussions
Evaluation:
Community feedback
Ontology fitness-for-purpose
Precision & recall?
25
26. Digital Enterprise Research Institute www.deri.ie
Problem
Possible Uses of a Knowledge Representation
Concrete Examples
Some Current Directions
26
27. The Problem
Digital Enterprise Research Institute www.deri.ie
The Web is full of opinions & commentary.
A lot of it disagrees.
How do we learn from other people, when they
disagree?
27
28. My Approach
Digital Enterprise Research Institute www.deri.ie
Identify peoples’ views
Collect the explanations people give
Create a hypertext web of these views & explanations
28
34. Purpose-related keywords
Digital Enterprise Research Institute www.deri.ie
Knowledge
statistics
Values
truth
secret
Rhetoric
you can thank
Judgment/Opinion
eradicate
tough
rejecting
34
35. Purpose matters
Digital Enterprise Research Institute www.deri.ie
Knowledge-oriented discussions are straightforward
to reuse
Opinion-oriented discussion types may require
caveating or balancing
emotion makes a discussion more interesting
can also indicate the potential for bias.
35
36. Twitter: Standpoint
Digital Enterprise Research Institute www.deri.ie
Difference between cakes and biscuits? When stale,
cakes go hard, biscuits go soft. Hence Jaffa Cakes
are cakes. (Was official EU ruling).
View: Jaffa Cakes are cakes
Justification: official EU ruling; go hard when stale
36
Mixed-initiative Generate argument maps from conversations (Arvina, MAgtALO) Populate a knowledge base Maybe change your views
They don’t say how they extracted these – but they say Someone makes statement (1) Someone else gives (4) as a reason/premise for (1) Someone else gives (3) as an additional reason for (1) (2) Is a counterproposal with a range of supporting reasons === Icons: http://findicons.com/icon/27954/girl_5?id=27964# http://findicons.com/icon/27930/boy_8?id=27939# http://findicons.com/icon/27955/girl_4?id=27965#
Maximal consistent sets
detect the prevalence of knowledge, emotion, and values as a first approximation to the purpose. High sentiment and low sentiment messages can be found through sen- timent analysis [21], which we also use as a first indication of whether people agree and how strongly their views are expressed. Values are abstract qualities such as utility, beauty, respect, and patriotism; these can be found with gazetteers. Knowledge-based discussions often cite statistics, experts, and studies, which can be text-mined; they may also commonly use argumentation schemes such as expert opinion.
Detecting the purpose of the discussion… Using keywords and rhetorical analysis Provides context