200114 Hyunwook Lee

1. Attention Models in Graphs: A Survey
John B. Lee, Ryan A. Rossi, Sungchul Kim, Nesreen K. Ahmed, Eunyee Koh
Adobe Research
TKDD’2018

2. Contents
• Definition of basic keywords
• Overview
• Model description
• Criticisms/benefits

3. Definition of Basic keywords
2

4. Definition of Basic keywords
3

5. Definition of Basic keywords
4

6. Overview
5
• Three different taxonomies based on
 Problem setting – especially, based on
input/output format
 Type of attention
 Task/problem

7. Overview
6

8. AttentionWalks
• Input: Homogeneous graph
• Output: Node embedding
• Mechanism: Learn attention weights + Attention-guided walk
• DeepWalk + Attention
• Performance of DeepWalk is sensitive to context window size c
7
Sami Abu-El-Haija, Bryan Perozzi, Rami Al-Rfou, and Alex Alemi. 2017. Watch Your Step: Learning Graph
Embeddings Through Attention. In arXiv:1710.09599v1

9. GAKE
• Input: Homogeneous graph(knowledge graph)
• Output: Node embedding
• Mechanism: Learn attention weights
• Node context 계산 시 edge 정보도 사용
• 최종 node embedding 계산에 attention 사용
8
Jun Feng, Minlie Huang, Yang Yang, and Xiaoyan Zhu. 2016. GAKE: Graph Aware Knowledge
Embedding. In Proc. of COLING. 641–651

10. GAT
• Input: Homogeneous graph
• Output: Node embedding
• Mechanism: Learn attention weights
• GCN(Graph-ConvNet) + Attention
• Using multi-head attention
9Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua
Bengio. 2018. Graph Attention Networks. In Proc. of ICLR. 1–12.

11. AGNN
• Input: Homogeneous graph
• Output: Node embedding
• Mechanism: Similarity-based attention
• GCN + Attention
• Similar to GAT, but using cosine similarity to calculate attention
• No multi-head attention
10Kiran K. Thekumparampil, Chong Wang, Sewoong Oh, and Li-Jia Li. 2018. Attention-based Graph
Neural Network for Semi-supervised Learning. In arXiv:1803.03735v1.

12. PRML
• Input: Homogeneous graph
• Output: Edge embedding
• Mechanism: Learn attention weights
• Path-based feature learning 이후에
 1) 각 path마다 attention(중요 노드 선정)
 2) path끼리 attention을 통해 최종 embedding 생성
11Zhou Zhao, Ben Gao, Vicent W. Zheng, Deng Cai, Xiaofei He, and Yueting Zhuang. 2017. Link
Prediction via Ranking Metric Dual-level Attention Network Learning. In Proc. of IJCAI. 3525–3531.

13. EAGCN
• Input: Heterogeneous graph
• Output: Graph embedding/Node embedding
• Mechanism: Learn attention weights
• GCN + Attention
• Multi attention to handle heterogeneous graph
12Chao Shang, Qinqing Liu, Ko-Shin Chen, Jiangwen Sun, Jin Lu, Jinfeng Yi, and Jinbo Bi. 2018. Edge
Attention-based Multi-Relational Graph Convolutional Networks. In arXiv:1802.04944v1.

14. Modified-GAT
• Input: Homogeneous graph
• Output: Graph embedding
• Mechanism: Similarity-based attention
• GAT+FCN, attention 계산이 약간 변함
• GAT를 통해 만들어진 Node embedding을 FCN을 통해 graph embedding
으로 변환
13Seongok Ryu, Jaechang Lim, and Woo Youn Kim. 2018. Deeply learning molecular structure-
property relationships using graph attention neural network. In arXiv:1805.10988v2.

15. graph2seq
• Input: Homogeneous graph
• Output: Graph embedding
• Mechanism: Similarity-based attention
• In node embedding, they consider both forward/backward neighbor
• In attention, they use node embedding
14Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, Michael Witbrock, and Vadim Sheinin. 2018. Graph2Seq:
Graph to Sequence Learning with Attention-based Neural Networks. In arXiv:1804.00823v3.

16. GAM
• Input: Homogeneous graph
• Output: Graph embedding
• Mechanism: Learn attention weights + Attention-guided walk
• Use RNN in attention-guided walk
• Use multi agent
15John Boaz Lee, Ryan Rossi, and Xiangnan Kong. 2018. Graph Classification using Structural Attention. In Proc. of
KDD. 1–9.

17. RNNSearch, Att-NMT
• Input: Path
• Output: Graph embedding
• Mechanism: Similarity-based attention
• Use hidden as embedding, calculate attention on every hidden, with
similarity between target hidden and hiddens
• In Att-NMT, there are local attention
16
Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural Machine
Translation by Jointly Learning to Align and Translate. In Proc. of ICLR. 1–15
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to
Attention-based Neural Machine Translation. In Proc. of EMNLP. 1412–1421.

18. Hao Zhou, Tom Yang, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense
Knowledge Aware Conversation Generation with Graph Attention. In Proc. of IJCAI-ECAI. 1–7
CCM
• Input: Homogeneous Graph
• Output: Graph embedding/Hybrid embedding
• Mechanism: Learn attention weights
• Get input sequence and knowledge graph
• Use two graph attention/one seq2seq attention
15

19. JointD/E+SATT
• Input: Homogeneous graph
• Output: Hybrid embedding
• Mechanism: Similarity-based attention
15Xu Han, Zhiyuan Liu, and Maosong Sun. 2018. Neural Knowledge Acquisition via Mutual Attention Between
Knowledge Graph and Text. In Proc. of AAAI. 1–8

20. GRAM
• Input: Directed acyclic graph
• Output: Hybrid embedding
• Mechanism: Learn attention weights
• DAG의 ancestor node를 모두 사용해서 attention
15Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F. Stewart, and Jimeng Sun. 2017. GRAM: Graph-
based Attention Model for Healthcare Representation Learning. In Proc. of KDD. 787–795

21. Criticisms
• Embedding을 나누는 기준이 모호함
• Input을 Homogeneous 등을 나누려는 시도는 좋았으나, survey한 논문
이 special case와 Homogeneous밖에 없음
• Problem setting기준 구분에서 전체 모델 소개가 끝나고,
mechanism/task기준 구분에 내용이 없음  Taxonomy 자체가 필요한
지를 잘 모르겠음
15

22. Criticisms
• Embedding을 나누는 기준이 모호함
• Input을 Homogeneous 등을 나누려는 시도는 좋았으나, survey한 논문
이 special case와 Homogeneous밖에 없음
• Problem setting기준 구분에서 전체 모델 소개가 끝나고,
mechanism/task기준 구분에 내용이 없음  Taxonomy 자체가 필요한
지를 잘 모르겠음
15

23. Benefits
• Trajectory data 다룰 때 CCM은 써볼 수 있을 듯함
• Cora dataset을 사용한 논문(e.g. GAT)은 현재 진행중인 task에 도움이 될
듯  다시 읽어볼 듯
15

24. Thanks You
Any Questions?