NS - CUK Seminar: V.T.Hoang, Review on "Structured self-attention architecture for graph-level representation learning", 2020

1. Van Thuy Hoang
Dept. of Artificial Intelligence,
The Catholic University of Korea
hoangvanthuy90@gmail.com
X. Fan, M. Gong and Y. Xie et al. / Pattern Recognition
100 (2020) 107084

2. 2
Contributions
 a Structured Self-attention Architecture for graph level representation
based on a GNN variant
 The proposed architecture’s readout can be incorporated into any
existing node-level GNNs and provide effective features for graph-
level representation.
 Compared with the pooling readout, the proposed architecture
shows its superior performance.

3. 3
Problem
 The main limitation of GNN schemes currently used for graphlevel
representation is the lack of effective utilization of graph
representation information.
 The contribution of each node to the output representation in the
pooling method is consistent.

4. 4
Problem
 Graph Attention Networks (GATs) introduce the selfattention
mechanism to node-level classification of graph structure
 Considering the graph-level readout, GAT layer cannot be directly
used to aggregate node representations due to inconsistent output
targets.

5. 5
Graph neural networks
 For graph classification, the readout function ϒ aggregates node
features to obtain the entire graph’s representation g in the form of

6. 6
Graph attention network
 Graph attention network (GAT) incorporates the attention mechanism
into the graph aggregate function.
 The representations of each node are computed by a selfattention
strategy

7. 7
Graph isomorphism network.
 Graph isomorphism network (GIN) aims to develop a simple
architecture that is as powerful as the Weisfeiler–Lehman graph
isomorphism test.

8. 8
Mathematical model
 Two types of self-attention for graph
 Node-focused self-attention for node-level output
 Graph-focused self-attention for graph-level output
 Structured self-attention architecture

9. 9
Two types of self-attention propagation for graph.
 Node-focused self-attention aims to generate the node-level
representation vectors by aggregating neighbor nodes and graph-
level self-attention focuses on aggregating nodes from the whole
graph.

10. 10
The overview of Structured Self-attention Architecture
 The graph-level output is generated by the node-focused and graph-
focused self-attention.
 The layer-focused self-attention generates the final graph-level
representation by aggregating layer-wise representations.
 T represents the number of layers.

11. 11
Node-focused self-attention for node-level output
 node representations obtain more local information as the number
of layers increases.

12. 12
Graph-focused self-attention for graph-level output
 Graph-level output aims to aggregate node features to obtain the
entire graph’s representation.

13. 13
Structured self-attention architecture
 Then graph-focused self-attention is used to obtain graph-level

14. 14
Optimized structured self-attention architecture.
 ZZZ

15. 15
Comparison and discussion
 Benchmarks
 MUTAG PTC PROTEINS NCI1 REDDIT-B
 REDDIT-M5K IMDB-B IMDB-M COLLAB

16. 16
Comparison of the 10-fold cross validation
 ZZZ

17. 17
Conclusion
 a Structured Self-attention Architecture for graph-level representation
 a scaled dot-product self-attention for node-level representation
learning and then introduces a graph-focused self-attention to
generate graph-level representation
 The scaled dot-product self-attention is faster and space-efficient and
the graph-focused self-attention tends to focus on the most
influential part of graph nodes
 a layer-focused self-attention which aggregates layer-wise graphlevel
representations