Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization

•Descargar como PPTX, PDF•

0 recomendaciones•68 vistas

The document analyzes the explainability of GraphSum, an abstractive multi-document summarization model, by examining its attention weights. It finds that GraphSum's attention weights from later decoding layers correlate more strongly with the relevance of input text segments, improving explainability. It also finds that GraphSum performs better when using paragraphs rather than sentences as input for the news domain, as paragraphs aid structure rather than topic separation for news articles. The document concludes that attention weights and expert annotations may provide better insight into abstractive summarization than ROUGE scores alone.

Ciencias

Analysis of GraphSum’s Attention
Weights to Improve the
Explainability of Multi-Document
Summarization
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 1
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner,
J. Töllich and A. Scherp

Extractive vs. Abstractive MDS
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 2
Input Documents
Model
Summary
Model
Extractive:
Abstractive:

Abstractive Graph-based MDS
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 3
Documents
Model
Summary
Sentences
Explainability ?

Research Questions
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 4
Model1
Sentences Paragraphs
Model2
Quality?
Documents
Model
Summary
Explainability?

GraphSum
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 5
Source: Li et al. “Leveraging Graph to Improve Abstractive Multi-Document Summarization” (2020)

Textual Unit Comparison
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

Build TF-IDF
Graph
Train
GraphSum
Model
Evaluate
Performance
Approach for Textual Units Comparison
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 7
06.04.2022

Pre-Processing
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 8
EXTRACTION
TRUNCATION
/
PADDING
TF-IDF
GRAPH
Build TF-IDF
Graph
Train
GraphSum
Model
Evaluate
Performance

GraphSum Training Procedure
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 9
Build TF-IDF
Graph
Train
GraphSum
Model
Evaluate
Performance
 Architecture and hyper-parameters as suggested by
Li et. al “Leveraging Graph to Improve Abstractive Multi-Document Summarization” (2020)
 Use similarity graph generated by pre-processing
 Use multiple batch-sizes
 Same number of input tokens
 Train / validation / test split

ROUGE Score
 ROUGE-2: Overlapping bi-grams
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 10
 ROUGE-L: Longest common subsequence
 Final score based on F-score as proposed by
Chin-Yew Lin, “ROUGE: A Package for Automatic Evaluation of Summaries” (2004)
Reference Reference
Candidate Candidate
Build TF-IDF
Graph
Train
GraphSum
Model
Evaluate
Performance

Explainability Analysis
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

Approach for Explainability Improvement
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 12

Data Sets
MultiNews WikiSum
Sentence vs Paragraphs x
Explainability Analysis x x
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 13
MultiNews:
Human written news summaries from professionals (60.000 Documents)
WikiSum:
Wikipedia articles and their references as MDS task (2.3 Million Arcticles)

Results: Textual Unit Comparison
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

Sentences vs Paragraphs
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 15
MultiNews

Usage of Paragraphs in News Domain
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 16
MultiNews

Results: Explainability Analysis
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

Attention Weights Correlation
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 18
Decoding Layer Multi-Heads

Correlation between Attention Weights and Reference Metric
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 19
MultiNews
Layer 6 (High Correlation)
Reference Metric
Attention
Weights
Reference Metric
Attention
Weights
Layer 3 (Low Correlation)

Positional Bias (MultiNews)
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 20

Conclusion
 Paragraphs perform better than sentences for news domain
 Paragraphs are used as structural aid, not for topic separation
 Other domains may show different behaviour
 Attention weights improve explainability of MDS
 Attention weights provide source origin information
 Latter decoding layers more suitable
 ROUGE score might not be fully applicable as metric for abstractive MDS
 ROUGE score not suitable for e.g., paraphrased sentences
 Expert annotated source information could provide better insights
06.04.2022
M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 21
Code available on GitHub: https://github.com/arnelochner/GBTBMDS

Más contenido relacionado

Más de Ansgar Scherp

We propose a pipeline for text extraction from infographics that makes use of a novel combination of data mining and computer vision techniques. The pipeline defines a sequence of steps to identify characters, cluster them into text lines, determine their rotation angle, and apply state-of-the-art OCR to recognize the text. In this paper, we formally define the pipeline and present its current implementation. In addition, we have conducted preliminary evaluations over a data corpus of 121 manually annotated infographics from a broad range of illustration types such as bar charts, pie charts, and line charts, maps, and others. We assess the results of our text extraction pipeline by comparing it with two baselines. Finally, we sketch an outline for future work and possibilities for improving the pipeline. - http://ceur-ws.org/Vol-1458/

Formalization and Preliminary Evaluation of a Pipeline for Text Extraction Fr...

Ansgar Scherp

Existing algorithms for signing graph data typically do not cover the whole signing process. In addition, they lack distinctive features such as signing graph data at different levels of granularity, iterative signing of graph data, and signing multiple graphs. In this paper, we introduce a novel framework for signing arbitrary graph data provided, e g., as RDF(S), Named Graphs, or OWL. We conduct an extensive theoretical and empirical analysis of the runtime and space complexity of different framework configurations. The experiments are performed on synthetic and real-world graph data of different size and different number of blank nodes. We investigate security issues, present a trust model, and discuss practical considerations for using our signing framework. We released a Java-based open source implementation of our software framework for iterative signing of arbitrary graph data provided, e. g., as RDF(S), Named Graphs, or OWL. The software framework is based on a formalization of different graph signing functions and supports different configurations. It is available in source code as well as pre-compiled as .jar-file. The graph signing framework exhibits the following unique features: - Signing graphs on different levels of granularity - Signing multiple graphs at once - Iterative signing of graph data for provenance tracking - Independence of the used language for encoding the graph (i. e., the signature does not break when changing the graph representation) The documentation of the software framework and its source code is available from: http://icp.it-risk.iwvi.uni-koblenz.de/wiki/Software_Framework_for_Signing_Graph_Data

A Framework for Iterative Signing of Graph Data on the Web

Ansgar Scherp

Manually selecting subsets of photos from large collections in order to present them to friends or colleagues or to print them as photo books can be a tedious task. Today, fully automatic approaches are at hand for supporting users. They make use of pixel information extracted from the images, analyze contextual information such as capture time and focal aperture, or use both to determine a proper subset of photos. However, these approaches miss the most important factor in the photo selection process: the user. The goal of our approach is to consider individual interests. By recording and analyzing gaze information from the user's viewing photo collections, we obtain information on user's interests and use this information in the creation of personal photo selections. In a controlled experiment with 33 participants, we show that the selections can be significantly improved over a baseline approach by up to 22% when taking individual viewing behavior into account. We also obtained significantly better results for photos taken at an event participants were involved in compared with photos from another event.

Smart photo selection: interpret gaze as personal interest

Ansgar Scherp

Events in Multimedia - Theory, Model, Application

Ansgar Scherp

Can you see it? Annotating Image Regions based on Users' Gaze Information

Ansgar Scherp

Linked open data - how to juggle with more than a billion triples

Ansgar Scherp

SchemEX -- Building an Index for Linked Open Data

Ansgar Scherp

SchemEX -- Building an Index for Linked Open Data

Ansgar Scherp

A Model of Events for Integrating Event-based Information in Complex Socio-te...

Ansgar Scherp

SchemEX - Creating the Yellow Pages for the Linked Open Data Cloud

Ansgar Scherp

strukt - A Pattern System for Integrating Individual and Organizational Knowl...

Ansgar Scherp

Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...

Ansgar Scherp

Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)

Ansgar Scherp

Más de Ansgar Scherp (13)

Formalization and Preliminary Evaluation of a Pipeline for Text Extraction Fr...

A Framework for Iterative Signing of Graph Data on the Web

Smart photo selection: interpret gaze as personal interest

Events in Multimedia - Theory, Model, Application

Can you see it? Annotating Image Regions based on Users' Gaze Information

Linked open data - how to juggle with more than a billion triples

SchemEX -- Building an Index for Linked Open Data

A Model of Events for Integrating Event-based Information in Complex Socio-te...

SchemEX - Creating the Yellow Pages for the Linked Open Data Cloud

strukt - A Pattern System for Integrating Individual and Organizational Knowl...

Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...

Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)

Último

Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS ESCORT SERVICE In Bhiwan...

Monika Rani

Dr. E. Muralinath_ Blood indices_clinical aspects

muralinath2

Velocity and Acceleration PowerPoint.ppt

RakeshMohan42

Introduction of DNA analysis in Forensic's .pptx

rohankumarsinghrore1

Locating and isolating a gene, FISH, GISH, Chromosome walking and jumping, te...

Silpa

Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....

muralinath2

Thyroid Physiology_Dr.E. Muralinath_ Associate Professor

muralinath2

CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA

Dr. TATHAGAT KHOBRAGADE

Ultrasound color Doppler imaging has been routinely used for the diagnosis of cardiovascular diseases, enabling real-time flow visualization through the Doppler effect. Yet, its inability to provide true flow velocity vectors due to its one-dimensional detection limits its efficacy. To overcome this limitation, various VFI schemes, including multi-angle beams, speckle tracking, and transverse oscillation, have been explored, with some already available commercially. However, many of these methods still rely on autocorrelation, which poses inherent issues such as underestimation, aliasing, and the need for large ensemble sizes. Conversely, speckle-tracking-based VFI enables lateral velocity estimation but suffers from significantly lower accuracy compared to axial velocity measurements. To address these challenges, we have presented a speckle-tracking-based VFI approach utilizing multi-angle ultrafast plane wave imaging. Our approach involves estimating axial velocity components projected onto individual steered plane waves, which are then combined to derive the velocity vector. Additionally, we've introduced a VFI visualization technique with high spatial and temporal resolutions capable of tracking flow particle trajectories. Simulation and flow phantom experiments demonstrate that the proposed VFI method outperforms both speckle-tracking-based VFI and autocorrelation VFI counterparts by at least a factor of three. Furthermore, in vivo measurements on carotid arteries using the Prodigy ultrasound scanner demonstrate the effectiveness of our approach compared to existing methods, providing a more robust imaging tool for hemodynamic studies. Learning objectives: - Understand fundamental limitations of color Doppler imaging. - Understand principles behind advanced vector flow imaging techniques. - Familiarize with the ultrasound speckle tracking technique and its implications in flow imaging. - Explore experiments conducted using multi-angle plane wave ultrafast imaging, specifically utilizing the pulse-sequence mode on a 128-channel ultrasound research platform.

(May 9, 2024) Enhanced Ultrafast Vector Flow Imaging (VFI) Using Multi-Angle ...

Scintica Instrumentation

An introduction on sequence tagged site mapping

adibshanto115

The computation of anti-derivatives is just an in-tellectual challenge, we know how to take deriv-atives, but … can we invert the process? We call this Computing the indefinite integral . In the last presentation we have seen a few indefinite integrals (we called them bricks), but they did not include the anti-derivative of many functions! We are going to try and do better !

COMPUTING ANTI-DERIVATIVES(Integration by SUBSTITUTION)

AkefAfaneh2

Grade 7 - Lesson 1 - Microscope and Its Functions

OrtegaSyrineMay

www.seribangash.com The Mariana Trench is one of the most remarkable geological features on Earth. Here are some details about it: Location: The Mariana Trench is located in the western Pacific Ocean, east of the Mariana Islands. It stretches for about 2,550 kilometers (1,580 miles) and is known as the deepest part of the world's oceans. Depth: The trench reaches incredible depths, with its deepest point known as the Challenger Deep, which plunges down to approximately 10,984 meters (36,037 feet) below sea level. To put this into perspective, if Mount Everest, the tallest mountain on Earth, were placed at the bottom of the Challenger Deep, its peak would still be over 2 kilometers (1.25 miles) underwater. Formation: The Mariana Trench was formed by the subduction of the Pacific Plate beneath the Mariana Plate. This process creates a deep trench as the heavier Pacific Plate is forced beneath the lighter Mariana Plate. Geological Features: The trench is characterized by steep, V-shaped valleys, and its walls are composed of highly compressed sedimentary rock. At the bottom of the trench, there are also large amounts of marine sediment. Pressure: The pressure at the bottom of the Mariana Trench is immense, reaching over 1,000 times the pressure at the surface. This extreme pressure creates a challenging environment for exploration and makes it difficult for organisms to survive. Exploration: Despite its extreme conditions, the Mariana Trench has been the subject of numerous scientific expeditions and explorations. One of the most famous explorations was the dive to the Challenger Deep by Swiss scientist Jacques Piccard and U.S. Navy Lieutenant Don Walsh in 1960. More recently, in 2012, filmmaker James Cameron made a solo dive to the bottom of the Challenger Deep in the Deepsea Challenger submersible. Biological Discoveries: Despite the harsh conditions, the Mariana Trench is home to a surprising variety of life forms, including unique species of deep-sea fish, crustaceans, and microbial life. Some organisms have adapted to survive in the extreme pressure and darkness of the trench. Environmental Importance: Studying the Mariana Trench provides valuable insights into the geology, biology, and oceanography of the deep sea. It also helps scientists better understand the processes that shape the Earth's crust and the distribution of life in the oceans. Conservation: Due to its remote location and extreme depths, the Mariana Trench has remained relatively untouched by human activity. However, there is growing concern about the potential impacts of deep-sea mining and pollution on this fragile ecosystem, highlighting the need for conservation efforts to protect this unique environment. https://seribangash.com/barber-shop-business-complete-guide-for-beginners/ https://seribangash.com/legend-virat-kohli-in-cricket-history/

The Mariana Trench remarkable geological features on Earth.pptx

seri bangash

module for grade 9 for distance learning

levieagacer

FAIRSpectra - Enabling the FAIRification of Analytical Science

Alex Henderson

Molecular markers are identifiable DNA sequences used to locate genes associated with specific traits or genetic conditions. A molecular marker is a specific gene fragment present at a specific position called ‘locus’ (pleural loci) in the genome of a cell. In the pool of unknown DNA or in a whole chromosome, these molecular markers help in identification of particular sequence of DNA at particular location.

Molecular markers- RFLP, RAPD, AFLP, SNP etc.

Silpa

GBSN - Microbiology (Unit 1)

Areesha Ahmad

Clean In Place(CIP).pptx .

Poonam Aher Patil

Porella : features, morphology, anatomy, reproduction etc.

Silpa

Proteomics: types, protein profiling steps etc.

Silpa

Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization

1. Analysis of GraphSum’s Attention Weights to Improve the Explainability of Multi-Document Summarization 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 1 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

2. Extractive vs. Abstractive MDS 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 2 Input Documents Model Summary Model Extractive: Abstractive:

3. Abstractive Graph-based MDS 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 3 Documents Model Summary Sentences Explainability ?

4. Research Questions 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 4 Model1 Sentences Paragraphs Model2 Quality? Documents Model Summary Explainability?

5. GraphSum 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 5 Source: Li et al. “Leveraging Graph to Improve Abstractive Multi-Document Summarization” (2020)

6. Textual Unit Comparison M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

7. Build TF-IDF Graph Train GraphSum Model Evaluate Performance Approach for Textual Units Comparison M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 7 06.04.2022

8. Pre-Processing 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 8 EXTRACTION TRUNCATION / PADDING TF-IDF GRAPH Build TF-IDF Graph Train GraphSum Model Evaluate Performance

9. GraphSum Training Procedure 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 9 Build TF-IDF Graph Train GraphSum Model Evaluate Performance  Architecture and hyper-parameters as suggested by Li et. al “Leveraging Graph to Improve Abstractive Multi-Document Summarization” (2020)  Use similarity graph generated by pre-processing  Use multiple batch-sizes  Same number of input tokens  Train / validation / test split

10. ROUGE Score  ROUGE-2: Overlapping bi-grams 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 10  ROUGE-L: Longest common subsequence  Final score based on F-score as proposed by Chin-Yew Lin, “ROUGE: A Package for Automatic Evaluation of Summaries” (2004) Reference Reference Candidate Candidate Build TF-IDF Graph Train GraphSum Model Evaluate Performance

11. Explainability Analysis M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

12. Approach for Explainability Improvement 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 12

13. Data Sets MultiNews WikiSum Sentence vs Paragraphs x Explainability Analysis x x 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 13 MultiNews: Human written news summaries from professionals (60.000 Documents) WikiSum: Wikipedia articles and their references as MDS task (2.3 Million Arcticles)

14. Results: Textual Unit Comparison M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

15. Sentences vs Paragraphs 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 15 MultiNews

16. Usage of Paragraphs in News Domain 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 16 MultiNews

17. Results: Explainability Analysis M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp

18. Attention Weights Correlation 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 18 Decoding Layer Multi-Heads

19. Correlation between Attention Weights and Reference Metric 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 19 MultiNews Layer 6 (High Correlation) Reference Metric Attention Weights Reference Metric Attention Weights Layer 3 (Low Correlation)

20. Positional Bias (MultiNews) 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 20

21. Conclusion  Paragraphs perform better than sentences for news domain  Paragraphs are used as structural aid, not for topic separation  Other domains may show different behaviour  Attention weights improve explainability of MDS  Attention weights provide source origin information  Latter decoding layers more suitable  ROUGE score might not be fully applicable as metric for abstractive MDS  ROUGE score not suitable for e.g., paraphrased sentences  Expert annotated source information could provide better insights 06.04.2022 M.L. Hickmann, F. Wurzberger, M. Hoxhalli, A. Lochner, J. Töllich and A. Scherp 21 Code available on GitHub: https://github.com/arnelochner/GBTBMDS

Notas del editor

Paragraphs: - Leveraging inter-paragraph relations can provide the model additional information for detecting contextual relations between topics. Sentences: - Our rationale is that with sentences as textual units, the graph structure represents inter-sentence relations, which may provide more detailed information within topics and thus may improve the results.
Batch Sizes GraphSum Model Hyperparamter as proposed by Li et al
Use tokenzier for extraction Same number of tokens
Wir haben ROUGE Scores als referenz verwendet Pearson Correlation
WikiSum nicht für Snetence vs Paragraphs aus resource limitations
Averaged Runs
Multi News Example
Basierend auf diesen Erkentnissen haben wir die Attention weights der Multi-heads aggregiert im weiteren Vorgehen
ROUGE Score ist Reference metric

Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization

Recomendados

Recomendados

Más contenido relacionado

Más de Ansgar Scherp

Más de Ansgar Scherp (13)

Último

Último (20)

Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization

Notas del editor