SlideShare una empresa de Scribd logo
1 de 38
Descargar para leer sin conexión
Towards Explainable
Fact Checking
Isabelle Augenstein
augenstein@di.ku.dk
@IAugenstein
http://isabelleaugenstein.github.io/
DIKU Business Club
18 March 2021
Overview of Today’s Talk
• Fact checking – what is it and why do we need it?
• False information online
• Content-based automatic fact checking
• Explainability – what is it and why do we need it?
• Making the right predictions for the right reasons
• Model training pipeline
• Explainable fact checking – some first solutions
• Rationale selection
• Generating free-text explanations
• Wrap-up
Fact Checking
3
Types of False Information
https://ec.europa.eu/info/live-work-travel-eu/coronavirus-response/fighting-disinformation/tackling-coronavirus-disinformation_en
Types of False Information
Types of False Information
http://www.contentrow.com/tools/link-bait-title-generator
Types of False Information
https://rokokoposten.dk/2021/02/23/mars-indfoerer-indrejseforbud-efter-fund-af-britisk-virusvariant/
Types of False Information
Types of False Information
• Disinformation:
• Intentionally false, spread deliberately
• Misinformation:
• Unintentionally false information
• Clickbait:
• Exaggerating information and under-delivering it
• Satire:
• Intentionally false for humorous purposes
• Biased Reporting:
• Reporting only some of the facts to serve an agenda
Types of False Information
• Disinformation:
• Intentionally false, spread deliberately
• Misinformation:
• Unintentionally false information
• Clickbait:
• Exaggerating information and under-delivering it
• Satire:
• Intentionally false for humorous purposes
• Biased Reporting:
• Reporting only some of the facts to serve an agenda
Full Fact Checking Pipeline
Claim Check-
Worthiness Detection
Evidence Document
Retrieval
Stance Detection /
Textual Entailment
Veracity Prediction
“Immigrants are a drain
on the economy”
not check-worthy
check-worthy
“Immigrants are a drain
on the economy”
“Immigrants are a drain on the
economy”, “EU immigrants have a
positive impact on public finances”
positive
negative
neutral
true
false
not enough info
“Immigrants are a drain
on the economy”
Explainability
12
Explainability – what is it and why do we need it?
Architect Trainer End User
Terminology borrowed from Strobelt et al. (2018), “LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent
neural networks. IEEE transactions on visualization and computer graphics.”
true
Explainability – what is it and why do we need it?
Claim: “In the COVID-19 crisis, ‘only 20%
of African Americans had jobs where they
could work from home.’”
Evidence: “20% of black workers said they
could work from home in their primary
job, compared to 30% of white workers.”
Claim: “Children don’t seem to be getting
this virus.”
Evidence: “There have been no reported
incidents of infection in children.”
Claim: “Taylor Swift had a fatal car
accident.”
Reason: overfitting to spurious patterns
(celebrity death hoaxes are common)
Claim: “Michael Jackson is still alive,
appears in daughter’s selfie.”
Reason: overfitting to spurious patterns
(celebrity death hoaxes are common)
Right prediction Wrong prediction
Right reasons
Wrong reasons
Explainability – what is it and why do we need it?
Architect Trainer End User
?
true
Explainability – what is it and why do we need it?
Architect Trainer End User
?
true
Explainability – what is it and why do we need it?
Architect Trainer End User
?
true
Towards
Explainable
Fact Checking
18
Explanation via Rationale Selection -- Sentences
Claim: The Faroe Islands are no longer part of a kingdom.
Evidence document: The Faroe Islands, also called the Faeroes, is an
archipelago between the Norwegian Sea and the North Atlantic, about halfway
between Norway and Iceland, 200 mi north-northwest of Scotland.
The Islands are an autonomous country within the Kingdom of Denmark.
Its area is about 1,400 km2 with a population of 50,030 in April 2017.
(…)
Label: refutes
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal (2018). FEVER: a
large-scale dataset for Fact Extraction and VERification. In Proc. of NAACL.
https://arxiv.org/abs/1803.05355
Explanation via Rationale Selection -- Sentences
Claim: The Faroe Islands are no longer part of a kingdom.
Evidence document: The Faroe Islands, also called the Faeroes, is an
archipelago between the Norwegian Sea and the North Atlantic, about halfway
between Norway and Iceland, 200 mi north-northwest of Scotland.
The Islands are an autonomous country within the Kingdom of Denmark.
Its area is about 1,400 km2 with a population of 50,030 in April 2017.
(…)
Label: refutes
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal (2018). FEVER: a
large-scale dataset for Fact Extraction and VERification. In Proc. of NAACL.
https://arxiv.org/abs/1803.05355
Explanation via Rationale Selection -- Words
Hypothesis: An adult dressed in black holds a stick.
Premise: An adult is walking away empty-handedly.
Label: contradiction
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, Phil Blunsom (2018). e-SNLI: Natural Language
Inference with Natural Language Explanations. In Proc. of NeurIPS.
https://proceedings.neurips.cc/paper/2018/hash/4c7a167bb329bd92580a99ce422d6fa6-Abstract.html
Explanation via Rationale Selection -- Words
Hypothesis: An adult dressed in black holds a stick.
Premise: An adult is walking away empty-handedly.
Label: contradiction
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, Phil Blunsom (2018). e-SNLI: Natural Language
Inference with Natural Language Explanations. In Proc. of NeurIPS.
https://proceedings.neurips.cc/paper/2018/hash/4c7a167bb329bd92580a99ce422d6fa6-Abstract.html
Post-Hoc Explainability Methods via Rationale Selection
Example from Twitter Sentiment Extraction (TSE) dataset
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein (2020). A Diagnostic Study of Explainability
Techniques for Text Classification. In EMNLP. https://www.aclweb.org/anthology/2020.emnlp-main.263/
• Agreement with Human Rationales (HA)
• Degree of overlap between human and automatic saliency scores
• Confidence Indication (CI)
• Predictive power of produced explanations for model’s confidence
• Faithfulness (F)
• Mask most salient tokens, measure drop in performance
• Rationale Consistency (RC)
• Difference between explanations for models trained with different
random seeds, with model with random weights
• Dataset Consistency (DC)
• Difference between explanations for similar instances
Post-Hoc Explainability Methods for via Rationale Selection:
Diagnostic Properties
Post-Hoc Explainability Methods via Rationale Selection:
Selected Results
Spider chart for Transformer model on e-SNLI
HA: Agreement with
human rationales
CI: Confidence indication
F: Faithfulness
RC: Rationale Consistency
DC: Dataset Consistency
Post-Hoc Explainability Methods via Rationale Selection:
Aggregated Results
0,4
0,45
0,5
0,55
0,6
0,65
0,7
Random
ShapSam
pl
LIM
E
O
cclusion
Saliency_u
Saliency_l2
InputXGrad_u
InputXGrad_l2
GuidedBP_u
GuidedBP_l2
Mean of diagnostic property measures for e-SNLI
Transformer CNN LSTM
Generating Free Text Explanations
https://www.politifact.com/factchecks/2020/apr/30/donald-trump/trumps-boasts-testing-spring-cherry-picking-data/
Ruling
Comments
Justification
https://www.politifact.com/factchecks/2020/apr/30/donald-trump/trumps-boasts-testing-spring-cherry-picking-data/
Generating Free Text Explanations
Generating Free Text Explanations
Claim:
We’ve tested more than every country
combined.
Ruling Comments:
Responding to weeks of criticism over his administration’s
COVID-19 response, President Donald Trump claimed at a
White House briefing that the United States has well
surpassed other countries in testing people for the virus.
"We’ve tested more than every country combined,"
Trump said April 27 […] We emailed the White House for
comment but never heard back, so we turned to the data.
Trump’s claim didn’t stand up to scrutiny.
In raw numbers, the United States has tested more
people than any other individual country — but
nowhere near more than "every country combined" or,
as he said in his tweet, more than "all major countries
combined.”[…] The United States has a far bigger
population than many of the "major countries" Trump often
mentions. So it could have run far more tests but still
have a much larger burden ahead than do nations like
Germany, France or Canada.[…]
Joint Model
Veracity Label
Justification/
Explanation
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein (2020). Generating Fact Checking
Explanations. In ACL. https://www.aclweb.org/anthology/2020.acl-main.656/
Generating Free Text Explanations:
(Manual) Evaluation Measures
• Explanation Quality
• Coverage. The explanation contains important, salient information
and does not miss any points important to the fact check.
• Non-redundancy. The explanation does not contain any information
that is redundant/repeated/not relevant to the claim or the fact
check.
• Non-contradiction. The explanation does not contain any pieces of
information contradictory to the claim or the fact check.
• Overall. Overall explanation quality.
• Explanation Informativeness. Veracity label for a claim
provided based on the automatically generated
explanations only.
Wrap-Up
31
• Why explainability?
• understanding if a model is right for the right reasons
• Generated explanations can help users understand:
• inner workings of a model (model understanding)
• how a model arrived at a prediction (decision understanding)
• Explainability can enable:
• human-in-the-loop model development
• human-in-the-loop data selection
Overall Take-Aways
• Caveats:
• There can be more than one correct explanation
• Different explainability methods provide different explanations
• Most work on simple (image/text) classification tasks
• Different streams of explainability methods have different
benefits and downsides
• Perform well w.r.t. different properties
• Requirements for annotated training data
• Joint vs. post-hoc explanation generation
• Black box vs. white box
• Hypothesis testing vs. bottom-up understanding
• One-time analysis vs. continuous monitoring
Overall Take-Aways
Where to from here?
• Making explanations useful to model trainers and
end users
• Understanding how to interpret different explanations
• Combination of model and decision understanding
• Different explanation granularity for different users (e.g. content
moderator vs. online platform user)
• Some research on generating explanations, relatively little
work on understanding in what context they are useful
• (Automatically) evaluating explanations
• Human-in-the-loop development
Thank you!
isabelleaugenstein.github.io
augenstein@di.ku.dk
@IAugenstein
github.com/isabelleaugenstein
18/03/2021 35
Thanks to my PhD students and colleagues!
36
CopeNLU
https://copenlu.github.io/
Pepa Atanasova Dustin Wright Jakob Grue Simonsen Christina Lioma
Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, Alexander M. Rush.
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent
Neural Networks. In Proceedings of IEEE TVCG 2018.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal.
FEVER: a large-scale dataset for Fact Extraction and VERification. In
Proceedings. of NAACL 2018.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, Phil Blunsom. e-
SNLI: Natural Language Inference with Natural Language Explanations. In
Proceedings of NeurIPS 2018.
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein. A
Diagnostic Study of Explainability Techniques for Text Classification. In
Proceedings of EMNLP 2020.
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein.
Generating Fact Checking Explanations. In Proceedings of ACL 2020.
Referenced Papers
Pepa Atanasova*, Dustin Wright*, Isabelle Augenstein. Generating Label
Cohesive and Well-Formed Adversarial Claims. EMNLP 2020.
Nils Rethmeier, Vageesh Kumar Saxena, Isabelle Augenstein. TX-Ray:
Quantifying and Explaining Model-Knowledge Transfer in (Un)Supervised NLP.
In Proceedings of the Conference on UAI 2020.
Wojciech Ostrowski, Arnav Arora, Pepa Atanasova, Isabelle Augenstein. Multi-
Hop Fact Checking of Political Claims. Preprint, 2020.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima,
Casper Hansen, Christian Hansen and Jakob Grue Simonsen. MultiFC: A Real-
World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims. In
Proceedings of EMNLP 2019.
Liesbeth Allein, Isabelle Augenstein, Marie-Francine Moens. Time-Aware
Evidence Ranking for Fact-Checking. Preprint, 2020.
*equal contributions
Other Relevant Papers

Más contenido relacionado

La actualidad más candente

Stop the madness - Never doubt the quality of BI again using Data Governance
Stop the madness - Never doubt the quality of BI again using Data GovernanceStop the madness - Never doubt the quality of BI again using Data Governance
Stop the madness - Never doubt the quality of BI again using Data GovernanceMary Levins, PMP
 
SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...
SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...
SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...Will Critchlow
 
Opinion-based Article Ranking for Information Retrieval Systems: Factoids and...
Opinion-based Article Ranking for Information Retrieval Systems: Factoids and...Opinion-based Article Ranking for Information Retrieval Systems: Factoids and...
Opinion-based Article Ranking for Information Retrieval Systems: Factoids and...Koray Tugberk GUBUR
 
Artificial Intelligence in Education: Ethical Futures
Artificial Intelligence in Education: Ethical FuturesArtificial Intelligence in Education: Ethical Futures
Artificial Intelligence in Education: Ethical FuturesRobert Farrow
 
Automating Google Lighthouse
Automating Google LighthouseAutomating Google Lighthouse
Automating Google LighthouseHamlet Batista
 
Presentation statista corporate-account_22-03c
Presentation statista corporate-account_22-03cPresentation statista corporate-account_22-03c
Presentation statista corporate-account_22-03cStatista
 
The Ultimate Google Indexing Session
The Ultimate Google Indexing SessionThe Ultimate Google Indexing Session
The Ultimate Google Indexing SessionOnely
 
Antifragility in Digital Marketing
Antifragility in Digital MarketingAntifragility in Digital Marketing
Antifragility in Digital MarketingElias Dabbas
 
Brighton Y U No Reply_Bibi the Link Builder.pdf
Brighton Y U No Reply_Bibi the Link Builder.pdfBrighton Y U No Reply_Bibi the Link Builder.pdf
Brighton Y U No Reply_Bibi the Link Builder.pdfBibi the Link Builder
 
Semantic seo and the evolution of queries
Semantic seo and the evolution of queriesSemantic seo and the evolution of queries
Semantic seo and the evolution of queriesBill Slawski
 
You Don't Know SEO
You Don't Know SEOYou Don't Know SEO
You Don't Know SEOMichael King
 
SEO by Hypothesis
SEO by HypothesisSEO by Hypothesis
SEO by HypothesisTom Anthony
 
Brighton SEO 2022: How to ideate to land links in publications that really im...
Brighton SEO 2022: How to ideate to land links in publications that really im...Brighton SEO 2022: How to ideate to land links in publications that really im...
Brighton SEO 2022: How to ideate to land links in publications that really im...EvaCheng23
 
10 Must-HAve GA4 Reports for SEO - Brighton SEO Apr 2023
10 Must-HAve GA4 Reports for SEO - Brighton SEO Apr 202310 Must-HAve GA4 Reports for SEO - Brighton SEO Apr 2023
10 Must-HAve GA4 Reports for SEO - Brighton SEO Apr 2023AccuraCast
 
Der wissenschaftliche Dienst in der Digitalen Bibliothek. Was kommt nach dem ...
Der wissenschaftliche Dienst in der Digitalen Bibliothek. Was kommt nach dem ...Der wissenschaftliche Dienst in der Digitalen Bibliothek. Was kommt nach dem ...
Der wissenschaftliche Dienst in der Digitalen Bibliothek. Was kommt nach dem ...Dr. Achim Bonte
 
Building Data Products with BigQuery for PPC and SEO (SMX 2022)
Building Data Products with BigQuery for PPC and SEO (SMX 2022)Building Data Products with BigQuery for PPC and SEO (SMX 2022)
Building Data Products with BigQuery for PPC and SEO (SMX 2022)Christopher Gutknecht
 
Suomalaisten koululaisten digitaalisen median käyttö
Suomalaisten koululaisten digitaalisen median käyttöSuomalaisten koululaisten digitaalisen median käyttö
Suomalaisten koululaisten digitaalisen median käyttöHarto Pönkä
 

La actualidad más candente (20)

Stop the madness - Never doubt the quality of BI again using Data Governance
Stop the madness - Never doubt the quality of BI again using Data GovernanceStop the madness - Never doubt the quality of BI again using Data Governance
Stop the madness - Never doubt the quality of BI again using Data Governance
 
SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...
SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...
SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...
 
Predictimands
PredictimandsPredictimands
Predictimands
 
Opinion-based Article Ranking for Information Retrieval Systems: Factoids and...
Opinion-based Article Ranking for Information Retrieval Systems: Factoids and...Opinion-based Article Ranking for Information Retrieval Systems: Factoids and...
Opinion-based Article Ranking for Information Retrieval Systems: Factoids and...
 
Artificial Intelligence in Education: Ethical Futures
Artificial Intelligence in Education: Ethical FuturesArtificial Intelligence in Education: Ethical Futures
Artificial Intelligence in Education: Ethical Futures
 
Automating Google Lighthouse
Automating Google LighthouseAutomating Google Lighthouse
Automating Google Lighthouse
 
Presentation statista corporate-account_22-03c
Presentation statista corporate-account_22-03cPresentation statista corporate-account_22-03c
Presentation statista corporate-account_22-03c
 
The Ultimate Google Indexing Session
The Ultimate Google Indexing SessionThe Ultimate Google Indexing Session
The Ultimate Google Indexing Session
 
Antifragility in Digital Marketing
Antifragility in Digital MarketingAntifragility in Digital Marketing
Antifragility in Digital Marketing
 
Brighton Y U No Reply_Bibi the Link Builder.pdf
Brighton Y U No Reply_Bibi the Link Builder.pdfBrighton Y U No Reply_Bibi the Link Builder.pdf
Brighton Y U No Reply_Bibi the Link Builder.pdf
 
Semantic seo and the evolution of queries
Semantic seo and the evolution of queriesSemantic seo and the evolution of queries
Semantic seo and the evolution of queries
 
Data Cleansing
Data CleansingData Cleansing
Data Cleansing
 
You Don't Know SEO
You Don't Know SEOYou Don't Know SEO
You Don't Know SEO
 
SEO by Hypothesis
SEO by HypothesisSEO by Hypothesis
SEO by Hypothesis
 
Brighton SEO 2022: How to ideate to land links in publications that really im...
Brighton SEO 2022: How to ideate to land links in publications that really im...Brighton SEO 2022: How to ideate to land links in publications that really im...
Brighton SEO 2022: How to ideate to land links in publications that really im...
 
10 Must-HAve GA4 Reports for SEO - Brighton SEO Apr 2023
10 Must-HAve GA4 Reports for SEO - Brighton SEO Apr 202310 Must-HAve GA4 Reports for SEO - Brighton SEO Apr 2023
10 Must-HAve GA4 Reports for SEO - Brighton SEO Apr 2023
 
Data visualization
Data visualizationData visualization
Data visualization
 
Der wissenschaftliche Dienst in der Digitalen Bibliothek. Was kommt nach dem ...
Der wissenschaftliche Dienst in der Digitalen Bibliothek. Was kommt nach dem ...Der wissenschaftliche Dienst in der Digitalen Bibliothek. Was kommt nach dem ...
Der wissenschaftliche Dienst in der Digitalen Bibliothek. Was kommt nach dem ...
 
Building Data Products with BigQuery for PPC and SEO (SMX 2022)
Building Data Products with BigQuery for PPC and SEO (SMX 2022)Building Data Products with BigQuery for PPC and SEO (SMX 2022)
Building Data Products with BigQuery for PPC and SEO (SMX 2022)
 
Suomalaisten koululaisten digitaalisen median käyttö
Suomalaisten koululaisten digitaalisen median käyttöSuomalaisten koululaisten digitaalisen median käyttö
Suomalaisten koululaisten digitaalisen median käyttö
 

Similar a Towards Explainable Fact Checking (DIKU Business Club presentation)

Towards Explainable Fact Checking
Towards Explainable Fact CheckingTowards Explainable Fact Checking
Towards Explainable Fact CheckingIsabelle Augenstein
 
[Slides] What Do We Do with All This Big Data by Altimeter Group
[Slides] What Do We Do with All This Big Data by Altimeter Group[Slides] What Do We Do with All This Big Data by Altimeter Group
[Slides] What Do We Do with All This Big Data by Altimeter GroupAltimeter, a Prophet Company
 
Graphic Representation Grading GuideCOMTM541 Version 22.docx
Graphic Representation Grading GuideCOMTM541 Version 22.docxGraphic Representation Grading GuideCOMTM541 Version 22.docx
Graphic Representation Grading GuideCOMTM541 Version 22.docxwhittemorelucilla
 
Big Data and AI in Fighting Against COVID-19
Big Data and AI in Fighting Against COVID-19Big Data and AI in Fighting Against COVID-19
Big Data and AI in Fighting Against COVID-19Bill Liu
 
Big Data and AI for Covid-19
Big Data and AI for Covid-19Big Data and AI for Covid-19
Big Data and AI for Covid-19Andrew Zhang
 
Suicide Survey Statistics
Suicide Survey StatisticsSuicide Survey Statistics
Suicide Survey StatisticsAngela Overton
 
Qualitative Research Analysis Of Irritable Bowel Syndrome...
Qualitative Research Analysis Of Irritable Bowel Syndrome...Qualitative Research Analysis Of Irritable Bowel Syndrome...
Qualitative Research Analysis Of Irritable Bowel Syndrome...Anna Shaw
 
Determined fake news
Determined fake newsDetermined fake news
Determined fake newsDueMonster
 
인공지능은 의료를 어떻게 혁신할 것인가 (ver 2)
인공지능은 의료를 어떻게 혁신할 것인가 (ver 2)인공지능은 의료를 어떻게 혁신할 것인가 (ver 2)
인공지능은 의료를 어떻게 혁신할 것인가 (ver 2)Yoon Sup Choi
 
Essay Writing Online. . Step-By-Step Guide to Essay Writing - ESL Buzz
Essay Writing Online. . Step-By-Step Guide to Essay Writing - ESL BuzzEssay Writing Online. . Step-By-Step Guide to Essay Writing - ESL Buzz
Essay Writing Online. . Step-By-Step Guide to Essay Writing - ESL BuzzKari Wilson
 
How To Write A Good Paragraph 1 Paragraph Wr
How To Write A Good Paragraph 1 Paragraph WrHow To Write A Good Paragraph 1 Paragraph Wr
How To Write A Good Paragraph 1 Paragraph WrAllison Koehn
 
AI and Social Justice: From Avoiding Harms to Positive Action
AI and Social Justice: From Avoiding Harms to Positive ActionAI and Social Justice: From Avoiding Harms to Positive Action
AI and Social Justice: From Avoiding Harms to Positive ActionAlan Dix
 
Verification Theory
Verification TheoryVerification Theory
Verification TheoryAshley Jones
 
Machine Learning, Data Mining, and
Machine Learning, Data Mining, and Machine Learning, Data Mining, and
Machine Learning, Data Mining, and butest
 
onlineidentitytheftfinal.docxby Larry RivesFILET IME.docx
onlineidentitytheftfinal.docxby Larry RivesFILET IME.docxonlineidentitytheftfinal.docxby Larry RivesFILET IME.docx
onlineidentitytheftfinal.docxby Larry RivesFILET IME.docxhopeaustin33688
 
Human-in-the-loop: the Web as Foundation for interdisciplinary Data Science M...
Human-in-the-loop: the Web as Foundation for interdisciplinary Data Science M...Human-in-the-loop: the Web as Foundation for interdisciplinary Data Science M...
Human-in-the-loop: the Web as Foundation for interdisciplinary Data Science M...Stefan Dietze
 
Owl Research - Fact Books Owl Theme Classroom, Ow
Owl Research - Fact Books Owl Theme Classroom, OwOwl Research - Fact Books Owl Theme Classroom, Ow
Owl Research - Fact Books Owl Theme Classroom, OwJenna Caldejon
 

Similar a Towards Explainable Fact Checking (DIKU Business Club presentation) (20)

Explainability for NLP
Explainability for NLPExplainability for NLP
Explainability for NLP
 
Towards Explainable Fact Checking
Towards Explainable Fact CheckingTowards Explainable Fact Checking
Towards Explainable Fact Checking
 
[Slides] What Do We Do with All This Big Data by Altimeter Group
[Slides] What Do We Do with All This Big Data by Altimeter Group[Slides] What Do We Do with All This Big Data by Altimeter Group
[Slides] What Do We Do with All This Big Data by Altimeter Group
 
Graphic Representation Grading GuideCOMTM541 Version 22.docx
Graphic Representation Grading GuideCOMTM541 Version 22.docxGraphic Representation Grading GuideCOMTM541 Version 22.docx
Graphic Representation Grading GuideCOMTM541 Version 22.docx
 
Big Data and AI in Fighting Against COVID-19
Big Data and AI in Fighting Against COVID-19Big Data and AI in Fighting Against COVID-19
Big Data and AI in Fighting Against COVID-19
 
Big Data and AI for Covid-19
Big Data and AI for Covid-19Big Data and AI for Covid-19
Big Data and AI for Covid-19
 
Suicide Survey Statistics
Suicide Survey StatisticsSuicide Survey Statistics
Suicide Survey Statistics
 
Qualitative Research Analysis Of Irritable Bowel Syndrome...
Qualitative Research Analysis Of Irritable Bowel Syndrome...Qualitative Research Analysis Of Irritable Bowel Syndrome...
Qualitative Research Analysis Of Irritable Bowel Syndrome...
 
A Lab Report On Academic Writing Essay
A Lab Report On Academic Writing EssayA Lab Report On Academic Writing Essay
A Lab Report On Academic Writing Essay
 
Determined fake news
Determined fake newsDetermined fake news
Determined fake news
 
인공지능은 의료를 어떻게 혁신할 것인가 (ver 2)
인공지능은 의료를 어떻게 혁신할 것인가 (ver 2)인공지능은 의료를 어떻게 혁신할 것인가 (ver 2)
인공지능은 의료를 어떻게 혁신할 것인가 (ver 2)
 
Essay Writing Online. . Step-By-Step Guide to Essay Writing - ESL Buzz
Essay Writing Online. . Step-By-Step Guide to Essay Writing - ESL BuzzEssay Writing Online. . Step-By-Step Guide to Essay Writing - ESL Buzz
Essay Writing Online. . Step-By-Step Guide to Essay Writing - ESL Buzz
 
Suggested Annotative Bibliography Essay
Suggested Annotative Bibliography EssaySuggested Annotative Bibliography Essay
Suggested Annotative Bibliography Essay
 
How To Write A Good Paragraph 1 Paragraph Wr
How To Write A Good Paragraph 1 Paragraph WrHow To Write A Good Paragraph 1 Paragraph Wr
How To Write A Good Paragraph 1 Paragraph Wr
 
AI and Social Justice: From Avoiding Harms to Positive Action
AI and Social Justice: From Avoiding Harms to Positive ActionAI and Social Justice: From Avoiding Harms to Positive Action
AI and Social Justice: From Avoiding Harms to Positive Action
 
Verification Theory
Verification TheoryVerification Theory
Verification Theory
 
Machine Learning, Data Mining, and
Machine Learning, Data Mining, and Machine Learning, Data Mining, and
Machine Learning, Data Mining, and
 
onlineidentitytheftfinal.docxby Larry RivesFILET IME.docx
onlineidentitytheftfinal.docxby Larry RivesFILET IME.docxonlineidentitytheftfinal.docxby Larry RivesFILET IME.docx
onlineidentitytheftfinal.docxby Larry RivesFILET IME.docx
 
Human-in-the-loop: the Web as Foundation for interdisciplinary Data Science M...
Human-in-the-loop: the Web as Foundation for interdisciplinary Data Science M...Human-in-the-loop: the Web as Foundation for interdisciplinary Data Science M...
Human-in-the-loop: the Web as Foundation for interdisciplinary Data Science M...
 
Owl Research - Fact Books Owl Theme Classroom, Ow
Owl Research - Fact Books Owl Theme Classroom, OwOwl Research - Fact Books Owl Theme Classroom, Ow
Owl Research - Fact Books Owl Theme Classroom, Ow
 

Más de Isabelle Augenstein

Beyond Fact Checking — Modelling Information Change in Scientific Communication
Beyond Fact Checking — Modelling Information Change in Scientific CommunicationBeyond Fact Checking — Modelling Information Change in Scientific Communication
Beyond Fact Checking — Modelling Information Change in Scientific CommunicationIsabelle Augenstein
 
Determining the Credibility of Science Communication
Determining the Credibility of Science CommunicationDetermining the Credibility of Science Communication
Determining the Credibility of Science CommunicationIsabelle Augenstein
 
Tracking False Information Online
Tracking False Information OnlineTracking False Information Online
Tracking False Information OnlineIsabelle Augenstein
 
What can typological knowledge bases and language representations tell us abo...
What can typological knowledge bases and language representations tell us abo...What can typological knowledge bases and language representations tell us abo...
What can typological knowledge bases and language representations tell us abo...Isabelle Augenstein
 
Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate ...
Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate ...Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate ...
Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate ...Isabelle Augenstein
 
Learning with limited labelled data in NLP: multi-task learning and beyond
Learning with limited labelled data in NLP: multi-task learning and beyondLearning with limited labelled data in NLP: multi-task learning and beyond
Learning with limited labelled data in NLP: multi-task learning and beyondIsabelle Augenstein
 
Learning to read for automated fact checking
Learning to read for automated fact checkingLearning to read for automated fact checking
Learning to read for automated fact checkingIsabelle Augenstein
 
SemEval 2017 Task 10: ScienceIE – Extracting Keyphrases and Relations from Sc...
SemEval 2017 Task 10: ScienceIE – Extracting Keyphrases and Relations from Sc...SemEval 2017 Task 10: ScienceIE – Extracting Keyphrases and Relations from Sc...
SemEval 2017 Task 10: ScienceIE – Extracting Keyphrases and Relations from Sc...Isabelle Augenstein
 
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...Isabelle Augenstein
 
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...Isabelle Augenstein
 
Machine Reading Using Neural Machines (talk at Microsoft Research Faculty Sum...
Machine Reading Using Neural Machines (talk at Microsoft Research Faculty Sum...Machine Reading Using Neural Machines (talk at Microsoft Research Faculty Sum...
Machine Reading Using Neural Machines (talk at Microsoft Research Faculty Sum...Isabelle Augenstein
 
Weakly Supervised Machine Reading
Weakly Supervised Machine ReadingWeakly Supervised Machine Reading
Weakly Supervised Machine ReadingIsabelle Augenstein
 
USFD at SemEval-2016 - Stance Detection on Twitter with Autoencoders
USFD at SemEval-2016 - Stance Detection on Twitter with AutoencodersUSFD at SemEval-2016 - Stance Detection on Twitter with Autoencoders
USFD at SemEval-2016 - Stance Detection on Twitter with AutoencodersIsabelle Augenstein
 
Distant Supervision with Imitation Learning
Distant Supervision with Imitation LearningDistant Supervision with Imitation Learning
Distant Supervision with Imitation LearningIsabelle Augenstein
 
Extracting Relations between Non-Standard Entities using Distant Supervision ...
Extracting Relations between Non-Standard Entities using Distant Supervision ...Extracting Relations between Non-Standard Entities using Distant Supervision ...
Extracting Relations between Non-Standard Entities using Distant Supervision ...Isabelle Augenstein
 
Information Extraction with Linked Data
Information Extraction with Linked DataInformation Extraction with Linked Data
Information Extraction with Linked DataIsabelle Augenstein
 
Lodifier: Generating Linked Data from Unstructured Text
Lodifier: Generating Linked Data from Unstructured TextLodifier: Generating Linked Data from Unstructured Text
Lodifier: Generating Linked Data from Unstructured TextIsabelle Augenstein
 
Relation Extraction from the Web using Distant Supervision
Relation Extraction from the Web using Distant SupervisionRelation Extraction from the Web using Distant Supervision
Relation Extraction from the Web using Distant SupervisionIsabelle Augenstein
 
Natural Language Processing for the Semantic Web
Natural Language Processing for the Semantic WebNatural Language Processing for the Semantic Web
Natural Language Processing for the Semantic WebIsabelle Augenstein
 
Seed Selection for Distantly Supervised Web-Based Relation Extraction
Seed Selection for Distantly Supervised Web-Based Relation ExtractionSeed Selection for Distantly Supervised Web-Based Relation Extraction
Seed Selection for Distantly Supervised Web-Based Relation ExtractionIsabelle Augenstein
 

Más de Isabelle Augenstein (20)

Beyond Fact Checking — Modelling Information Change in Scientific Communication
Beyond Fact Checking — Modelling Information Change in Scientific CommunicationBeyond Fact Checking — Modelling Information Change in Scientific Communication
Beyond Fact Checking — Modelling Information Change in Scientific Communication
 
Determining the Credibility of Science Communication
Determining the Credibility of Science CommunicationDetermining the Credibility of Science Communication
Determining the Credibility of Science Communication
 
Tracking False Information Online
Tracking False Information OnlineTracking False Information Online
Tracking False Information Online
 
What can typological knowledge bases and language representations tell us abo...
What can typological knowledge bases and language representations tell us abo...What can typological knowledge bases and language representations tell us abo...
What can typological knowledge bases and language representations tell us abo...
 
Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate ...
Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate ...Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate ...
Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate ...
 
Learning with limited labelled data in NLP: multi-task learning and beyond
Learning with limited labelled data in NLP: multi-task learning and beyondLearning with limited labelled data in NLP: multi-task learning and beyond
Learning with limited labelled data in NLP: multi-task learning and beyond
 
Learning to read for automated fact checking
Learning to read for automated fact checkingLearning to read for automated fact checking
Learning to read for automated fact checking
 
SemEval 2017 Task 10: ScienceIE – Extracting Keyphrases and Relations from Sc...
SemEval 2017 Task 10: ScienceIE – Extracting Keyphrases and Relations from Sc...SemEval 2017 Task 10: ScienceIE – Extracting Keyphrases and Relations from Sc...
SemEval 2017 Task 10: ScienceIE – Extracting Keyphrases and Relations from Sc...
 
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
 
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
1st Workshop for Women and Underrepresented Minorities (WiNLP) at ACL 2017 - ...
 
Machine Reading Using Neural Machines (talk at Microsoft Research Faculty Sum...
Machine Reading Using Neural Machines (talk at Microsoft Research Faculty Sum...Machine Reading Using Neural Machines (talk at Microsoft Research Faculty Sum...
Machine Reading Using Neural Machines (talk at Microsoft Research Faculty Sum...
 
Weakly Supervised Machine Reading
Weakly Supervised Machine ReadingWeakly Supervised Machine Reading
Weakly Supervised Machine Reading
 
USFD at SemEval-2016 - Stance Detection on Twitter with Autoencoders
USFD at SemEval-2016 - Stance Detection on Twitter with AutoencodersUSFD at SemEval-2016 - Stance Detection on Twitter with Autoencoders
USFD at SemEval-2016 - Stance Detection on Twitter with Autoencoders
 
Distant Supervision with Imitation Learning
Distant Supervision with Imitation LearningDistant Supervision with Imitation Learning
Distant Supervision with Imitation Learning
 
Extracting Relations between Non-Standard Entities using Distant Supervision ...
Extracting Relations between Non-Standard Entities using Distant Supervision ...Extracting Relations between Non-Standard Entities using Distant Supervision ...
Extracting Relations between Non-Standard Entities using Distant Supervision ...
 
Information Extraction with Linked Data
Information Extraction with Linked DataInformation Extraction with Linked Data
Information Extraction with Linked Data
 
Lodifier: Generating Linked Data from Unstructured Text
Lodifier: Generating Linked Data from Unstructured TextLodifier: Generating Linked Data from Unstructured Text
Lodifier: Generating Linked Data from Unstructured Text
 
Relation Extraction from the Web using Distant Supervision
Relation Extraction from the Web using Distant SupervisionRelation Extraction from the Web using Distant Supervision
Relation Extraction from the Web using Distant Supervision
 
Natural Language Processing for the Semantic Web
Natural Language Processing for the Semantic WebNatural Language Processing for the Semantic Web
Natural Language Processing for the Semantic Web
 
Seed Selection for Distantly Supervised Web-Based Relation Extraction
Seed Selection for Distantly Supervised Web-Based Relation ExtractionSeed Selection for Distantly Supervised Web-Based Relation Extraction
Seed Selection for Distantly Supervised Web-Based Relation Extraction
 

Último

March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 

Último (20)

March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 

Towards Explainable Fact Checking (DIKU Business Club presentation)

  • 1. Towards Explainable Fact Checking Isabelle Augenstein augenstein@di.ku.dk @IAugenstein http://isabelleaugenstein.github.io/ DIKU Business Club 18 March 2021
  • 2. Overview of Today’s Talk • Fact checking – what is it and why do we need it? • False information online • Content-based automatic fact checking • Explainability – what is it and why do we need it? • Making the right predictions for the right reasons • Model training pipeline • Explainable fact checking – some first solutions • Rationale selection • Generating free-text explanations • Wrap-up
  • 4. Types of False Information https://ec.europa.eu/info/live-work-travel-eu/coronavirus-response/fighting-disinformation/tackling-coronavirus-disinformation_en
  • 5. Types of False Information
  • 6. Types of False Information http://www.contentrow.com/tools/link-bait-title-generator
  • 7. Types of False Information https://rokokoposten.dk/2021/02/23/mars-indfoerer-indrejseforbud-efter-fund-af-britisk-virusvariant/
  • 8. Types of False Information
  • 9. Types of False Information • Disinformation: • Intentionally false, spread deliberately • Misinformation: • Unintentionally false information • Clickbait: • Exaggerating information and under-delivering it • Satire: • Intentionally false for humorous purposes • Biased Reporting: • Reporting only some of the facts to serve an agenda
  • 10. Types of False Information • Disinformation: • Intentionally false, spread deliberately • Misinformation: • Unintentionally false information • Clickbait: • Exaggerating information and under-delivering it • Satire: • Intentionally false for humorous purposes • Biased Reporting: • Reporting only some of the facts to serve an agenda
  • 11. Full Fact Checking Pipeline Claim Check- Worthiness Detection Evidence Document Retrieval Stance Detection / Textual Entailment Veracity Prediction “Immigrants are a drain on the economy” not check-worthy check-worthy “Immigrants are a drain on the economy” “Immigrants are a drain on the economy”, “EU immigrants have a positive impact on public finances” positive negative neutral true false not enough info “Immigrants are a drain on the economy”
  • 13. Explainability – what is it and why do we need it? Architect Trainer End User Terminology borrowed from Strobelt et al. (2018), “LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE transactions on visualization and computer graphics.” true
  • 14. Explainability – what is it and why do we need it? Claim: “In the COVID-19 crisis, ‘only 20% of African Americans had jobs where they could work from home.’” Evidence: “20% of black workers said they could work from home in their primary job, compared to 30% of white workers.” Claim: “Children don’t seem to be getting this virus.” Evidence: “There have been no reported incidents of infection in children.” Claim: “Taylor Swift had a fatal car accident.” Reason: overfitting to spurious patterns (celebrity death hoaxes are common) Claim: “Michael Jackson is still alive, appears in daughter’s selfie.” Reason: overfitting to spurious patterns (celebrity death hoaxes are common) Right prediction Wrong prediction Right reasons Wrong reasons
  • 15. Explainability – what is it and why do we need it? Architect Trainer End User ? true
  • 16. Explainability – what is it and why do we need it? Architect Trainer End User ? true
  • 17. Explainability – what is it and why do we need it? Architect Trainer End User ? true
  • 19. Explanation via Rationale Selection -- Sentences Claim: The Faroe Islands are no longer part of a kingdom. Evidence document: The Faroe Islands, also called the Faeroes, is an archipelago between the Norwegian Sea and the North Atlantic, about halfway between Norway and Iceland, 200 mi north-northwest of Scotland. The Islands are an autonomous country within the Kingdom of Denmark. Its area is about 1,400 km2 with a population of 50,030 in April 2017. (…) Label: refutes James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal (2018). FEVER: a large-scale dataset for Fact Extraction and VERification. In Proc. of NAACL. https://arxiv.org/abs/1803.05355
  • 20. Explanation via Rationale Selection -- Sentences Claim: The Faroe Islands are no longer part of a kingdom. Evidence document: The Faroe Islands, also called the Faeroes, is an archipelago between the Norwegian Sea and the North Atlantic, about halfway between Norway and Iceland, 200 mi north-northwest of Scotland. The Islands are an autonomous country within the Kingdom of Denmark. Its area is about 1,400 km2 with a population of 50,030 in April 2017. (…) Label: refutes James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal (2018). FEVER: a large-scale dataset for Fact Extraction and VERification. In Proc. of NAACL. https://arxiv.org/abs/1803.05355
  • 21. Explanation via Rationale Selection -- Words Hypothesis: An adult dressed in black holds a stick. Premise: An adult is walking away empty-handedly. Label: contradiction Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, Phil Blunsom (2018). e-SNLI: Natural Language Inference with Natural Language Explanations. In Proc. of NeurIPS. https://proceedings.neurips.cc/paper/2018/hash/4c7a167bb329bd92580a99ce422d6fa6-Abstract.html
  • 22. Explanation via Rationale Selection -- Words Hypothesis: An adult dressed in black holds a stick. Premise: An adult is walking away empty-handedly. Label: contradiction Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, Phil Blunsom (2018). e-SNLI: Natural Language Inference with Natural Language Explanations. In Proc. of NeurIPS. https://proceedings.neurips.cc/paper/2018/hash/4c7a167bb329bd92580a99ce422d6fa6-Abstract.html
  • 23. Post-Hoc Explainability Methods via Rationale Selection Example from Twitter Sentiment Extraction (TSE) dataset Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein (2020). A Diagnostic Study of Explainability Techniques for Text Classification. In EMNLP. https://www.aclweb.org/anthology/2020.emnlp-main.263/
  • 24. • Agreement with Human Rationales (HA) • Degree of overlap between human and automatic saliency scores • Confidence Indication (CI) • Predictive power of produced explanations for model’s confidence • Faithfulness (F) • Mask most salient tokens, measure drop in performance • Rationale Consistency (RC) • Difference between explanations for models trained with different random seeds, with model with random weights • Dataset Consistency (DC) • Difference between explanations for similar instances Post-Hoc Explainability Methods for via Rationale Selection: Diagnostic Properties
  • 25. Post-Hoc Explainability Methods via Rationale Selection: Selected Results Spider chart for Transformer model on e-SNLI HA: Agreement with human rationales CI: Confidence indication F: Faithfulness RC: Rationale Consistency DC: Dataset Consistency
  • 26. Post-Hoc Explainability Methods via Rationale Selection: Aggregated Results 0,4 0,45 0,5 0,55 0,6 0,65 0,7 Random ShapSam pl LIM E O cclusion Saliency_u Saliency_l2 InputXGrad_u InputXGrad_l2 GuidedBP_u GuidedBP_l2 Mean of diagnostic property measures for e-SNLI Transformer CNN LSTM
  • 27. Generating Free Text Explanations https://www.politifact.com/factchecks/2020/apr/30/donald-trump/trumps-boasts-testing-spring-cherry-picking-data/
  • 29. Generating Free Text Explanations Claim: We’ve tested more than every country combined. Ruling Comments: Responding to weeks of criticism over his administration’s COVID-19 response, President Donald Trump claimed at a White House briefing that the United States has well surpassed other countries in testing people for the virus. "We’ve tested more than every country combined," Trump said April 27 […] We emailed the White House for comment but never heard back, so we turned to the data. Trump’s claim didn’t stand up to scrutiny. In raw numbers, the United States has tested more people than any other individual country — but nowhere near more than "every country combined" or, as he said in his tweet, more than "all major countries combined.”[…] The United States has a far bigger population than many of the "major countries" Trump often mentions. So it could have run far more tests but still have a much larger burden ahead than do nations like Germany, France or Canada.[…] Joint Model Veracity Label Justification/ Explanation Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein (2020). Generating Fact Checking Explanations. In ACL. https://www.aclweb.org/anthology/2020.acl-main.656/
  • 30. Generating Free Text Explanations: (Manual) Evaluation Measures • Explanation Quality • Coverage. The explanation contains important, salient information and does not miss any points important to the fact check. • Non-redundancy. The explanation does not contain any information that is redundant/repeated/not relevant to the claim or the fact check. • Non-contradiction. The explanation does not contain any pieces of information contradictory to the claim or the fact check. • Overall. Overall explanation quality. • Explanation Informativeness. Veracity label for a claim provided based on the automatically generated explanations only.
  • 32. • Why explainability? • understanding if a model is right for the right reasons • Generated explanations can help users understand: • inner workings of a model (model understanding) • how a model arrived at a prediction (decision understanding) • Explainability can enable: • human-in-the-loop model development • human-in-the-loop data selection Overall Take-Aways
  • 33. • Caveats: • There can be more than one correct explanation • Different explainability methods provide different explanations • Most work on simple (image/text) classification tasks • Different streams of explainability methods have different benefits and downsides • Perform well w.r.t. different properties • Requirements for annotated training data • Joint vs. post-hoc explanation generation • Black box vs. white box • Hypothesis testing vs. bottom-up understanding • One-time analysis vs. continuous monitoring Overall Take-Aways
  • 34. Where to from here? • Making explanations useful to model trainers and end users • Understanding how to interpret different explanations • Combination of model and decision understanding • Different explanation granularity for different users (e.g. content moderator vs. online platform user) • Some research on generating explanations, relatively little work on understanding in what context they are useful • (Automatically) evaluating explanations • Human-in-the-loop development
  • 36. Thanks to my PhD students and colleagues! 36 CopeNLU https://copenlu.github.io/ Pepa Atanasova Dustin Wright Jakob Grue Simonsen Christina Lioma
  • 37. Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, Alexander M. Rush. LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks. In Proceedings of IEEE TVCG 2018. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal. FEVER: a large-scale dataset for Fact Extraction and VERification. In Proceedings. of NAACL 2018. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, Phil Blunsom. e- SNLI: Natural Language Inference with Natural Language Explanations. In Proceedings of NeurIPS 2018. Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein. A Diagnostic Study of Explainability Techniques for Text Classification. In Proceedings of EMNLP 2020. Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein. Generating Fact Checking Explanations. In Proceedings of ACL 2020. Referenced Papers
  • 38. Pepa Atanasova*, Dustin Wright*, Isabelle Augenstein. Generating Label Cohesive and Well-Formed Adversarial Claims. EMNLP 2020. Nils Rethmeier, Vageesh Kumar Saxena, Isabelle Augenstein. TX-Ray: Quantifying and Explaining Model-Knowledge Transfer in (Un)Supervised NLP. In Proceedings of the Conference on UAI 2020. Wojciech Ostrowski, Arnav Arora, Pepa Atanasova, Isabelle Augenstein. Multi- Hop Fact Checking of Political Claims. Preprint, 2020. Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen and Jakob Grue Simonsen. MultiFC: A Real- World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims. In Proceedings of EMNLP 2019. Liesbeth Allein, Isabelle Augenstein, Marie-Francine Moens. Time-Aware Evidence Ranking for Fact-Checking. Preprint, 2020. *equal contributions Other Relevant Papers