Graph-to-Sequence Learning using Gated Graph Neural Networks. [ACL'18] 論文紹介
1. 2019.05.07
NAIST ⾃自然⾔言語処理理学研究室
D3 Masayoshi Kondo
論論⽂文紹介-‐‑‒ About Graph Neural Networks@2019
Graph-‐‑‒to-‐‑‒Sequence Learning
using Gated Graph Neural Networks
ACLʼ’18
Daniel1 Beck Gholamreza2 Haffari Trevor Cohn1
1School of Computing and Information Systems
University of Melbourne, Australia
2Faculty of Information Technology
Monash University, Australia
3. 1. Introduction
2. Neural Graph-‐‑‒to-‐‑‒Sequence Model
3. Levi Graph Transformation
4. Generation from AMR Graphs
5. Syntax-‐‑‒based Neural Machine Translation
6. Related work
7. Discussion and Conclusion
4. 1. Introduction
2. Neural Graph-‐‑‒to-‐‑‒Sequence Model
3. Levi Graph Transformation
4. Generation from AMR Graphs
5. Syntax-‐‑‒based Neural Machine Translation
6. Related work
7. Discussion and Conclusion
5. 【背景】
多くのNLPの応⽤用は、グラフ構造から系列列への変換⼿手続きとして枠組み化
できる.
• ⾔言語⽣生成(language generation) :
a semantic graph -‐‑‒> a surface form
• ⽂文法構造を⽤用いる機械翻訳 (syntactic machine translation) :
a tree-‐‑‒annotated source sentence -‐‑‒> its translation.
< 具体例例 >
00 : Introduction
【課題/問題点】
< 先⾏行行研究例例 >
• Grammar based approaches : [Flanigan et al., 2016], [Jones et al., 2012]
• To transform a graph into a linearised form :
[Pourgamghani et al., 2016], [Konstas et al., 2017]
• グラフと表層項(surface tokens)の間のアラインメントにおいて、
grammar構築時のエラーがそのまま(アラインメントの⽅方へ)伝搬.
• linearisationでは、完全なグラフ構造や重要情報を考慮出来ない.
7. 1. Introduction
2. Neural Graph-‐‑‒to-‐‑‒Sequence Model
3. Levi Graph Transformation
4. Generation from AMR Graphs
5. Syntax-‐‑‒based Neural Machine Translation
6. Related work
7. Discussion and Conclusion
8. 00 : Neural Graph-‐‑‒to-‐‑‒Sequence Model
Input :
AMR Graph
9. 00 : Neural Graph-‐‑‒to-‐‑‒Sequence Model
2.1 Gated Graph Neural Networks (GGNNs)[Li et al., 2016]
• 添字tは、恐らく層数(t th-‐‑‒layer)を指す.
• GRUと似た構造だが、⾮非再帰型モデル.
• エッジラベル毎に重み⾏行行列列を備えているので、
エッジラベル数が多いグラフの場合はパラ
メータ爆発を引き起こす(エッジラベル数に
対して線形増加.)
G = V, E, LV , LE{ }
(υ,lυ )
(υi,υj,lυ )
グラフ :
ノード集合 :
エッジ集合 :
・Lとlは、ラベル集合 と 各ラベル.
・右図の添字uは、上記のviを⽰示す.
gating
mechanism
cv = cv
z
= cv
r
= Nv
−1
パラメータ c は、正則化の定数. output
誤字、正しくはt
[ Li et al.,2016]のモデルとの変更更点
1. バイアス項、リセットゲート、
アップデートゲートの追加.
2. ラベル重み⾏行行列列の層間での⾮非共有化.
3. 全ての隠れ状態ベクトルへの
リセットゲートの適⽤用.
4. 正則化の定数の導⼊入.
11. 1. Introduction
2. Neural Graph-‐‑‒to-‐‑‒Sequence Model
3. Levi Graph Transformation
4. Generation from AMR Graphs
5. Syntax-‐‑‒based Neural Machine Translation
6. Related work
7. Discussion and Conclusion
13. 1. Introduction
2. Neural Graph-‐‑‒to-‐‑‒Sequence Model
3. Levi Graph Transformation
4. Generation from AMR Graphs
5. Syntax-‐‑‒based Neural Machine Translation
6. Related work
7. Discussion and Conclusion
15. 00 : Generation from AMR Graphs
4.2 Results and analysis
• ベースラインのs2sモデルに圧倒的
に精度度で上回っている.
• ベースラインのs2sモデルは、
scope markerが必要だが、提案法
のg2sモデルは、scope markerを
⽤用いること無くベースラインに勝っ
ている.
• 提案法のg2sモデルは、⼤大規模なラ
ベル無しデータと⼩小規模訓練セット
を併⽤用する既存モデルの多くに対し
て良良い結果を⽰示している.
• 最⾼高精度度は、Konstas et al.2017の
KIYCZモデルである.
16. 00 : Generation from AMR Graphs
4.2 Results and analysis
提案法g2sモデルは、
overgenerationを避けられている.
17. 1. Introduction
2. Neural Graph-‐‑‒to-‐‑‒Sequence Model
3. Levi Graph Transformation
4. Generation from AMR Graphs
5. Syntax-‐‑‒based Neural Machine Translation
6. Related work
7. Discussion and Conclusion
18. 00 : Syntax-‐‑‒based Neural Machine Translation
5.1 Experimental setup
【 Data and Preprocessing 】
データセット:News Commentary V11 corpora
from the WMT16 translation task
前処理理⼿手続き:same data and settings from Bastings et al. (2017)
【 Models 】: AMR-‐‑‒generationとほとんど同じ 【 Evaluation 】
• BLEU
• sentence-‐‑‒level CHRF++
-‐‑‒ Eng : tokenised and parsed using SyntaxNet7.
-‐‑‒ German and Czech : texts are tokenised and split into subwords
using byte-‐‑‒pair encodings [Sennrich et al., 2016, BPE]]
(8000 merge operations).
-‐‑‒ Dependent trees + sequential connections
• GGNN encoderの次元は、dependent
treeのみの場合が512、その他は448.
• S2sモデルはdependent tree(の情
報)は⽤用いず、単語系列列のみ.
• ⽐比較⼿手法に、Phrase-‐‑‒Based
Statistical MT(PB-‐‑‒SMT)も⽤用いる.
AMR-‐‑‒generation taskと設定
は同じ.
19. 00 : Syntax-‐‑‒based Neural Machine Translation
• 右表のg2s+モデルは、
graph+sequenctial
informationを⽰示す.
• BoW+GCNモデルと提案
法g2sモデルは、モデル
が良良く似ているが、提案
法が勝っており、その⼤大
きな違いは、Levi graph
変換とエッジを隠れ状態
ベクトルとして扱う点で
ある.
• NMTタスクでは、AMR
⽣生成タスクと同じ構造の
モデルを利利⽤用したため、
NMTタスク⽤用にチューニ
ングすることでさらに提
案法は性能が伸びる(こ
とが期待される.)
Dependent tree + Seq Info
5.2 Result and analysis
20. 1. Introduction
2. Neural Graph-‐‑‒to-‐‑‒Sequence Model
3. Levi Graph Transformation
4. Generation from AMR Graphs
5. Syntax-‐‑‒based Neural Machine Translation
6. Related work
7. Discussion and Conclusion
21. 00 : Related work
【Graph-‐‑‒to-‐‑‒sequence modelling】
• Hyperedge Replacement Graph Grammars
[Drewes et al., 1997, HRGs]
• Parsing Graphs with Hyperedge Replacement Grammars
[Chiang et al., 2013]
• Semantics-‐‑‒Based Machine Translation with Hyperedge
Replacement Grammars [Jones et al., COLING'12]
• A Synchronous Hyperedge Replacement Grammar based
approach for AMR parsing [Peng et al., CoNLL'15]
22. 00 : Related work
【Neural networks for graphs】
• A New Model for Learning in Graph Domains
[Gori et al., IJCNN'05]
• The Graph Neural Network Model
[Scarselli et al., IEEE Trans, 2009]
• Gated Graph Sequence Neural Networks [Li et al., ICLR'16]
• Spectral Networks and Locally Connected Networks on Graphs
[Bruna et al., ICLR'14]
• Convolutional Networks on Graphs for Learning Molecular
Fingerprints [Duvenaud et al., NIPS'15]
• Semi-‐‑‒Supervised Classification with Graph Convolutional
Networks [Kipf and Welling, ICLR'17]
• Encoding Sentences with Graph Convolutional Networks for
Semantic Role Labeling [Marcheggiani and Titov, EMNLP'17]
• Modeling Relational Data with Graph Convolutional Networks
[Schlichtkrull et al., 2017]
23. 00 : Related work
【Applications】
• Generation from Abstract Meaning Representation using Tree Transducers
[Flanigan et al., NAACL'16]
• AMR-‐‑‒to-‐‑‒text Generation with Synchronous Node Replacement Grammar [Song et al., ACL'17]
• Generating English from Abstract Meaning Representations [Pourdamghani et al., INLG'16]
• Neural AMR: Sequence-‐‑‒to-‐‑‒Sequence Models for Parsing and Generation
[Konstas et al., ACL'17]
• Stochastic inversion transduction grammars and bilingual parsing of parallel corpora
[Wu, Computational Linguistics 1997]
• A Syntax-‐‑‒based Statistical Translation Model [Yamada and Knight, ACL'01]
• Whatʼ’s in a translation rule ? [Galley et al., NAACL'04]
• Tree-‐‑‒to-‐‑‒string alignment template for statistical machine translation [Liu et al., ACL'06]
• Graph Convolutional Encoders for Syntax-‐‑‒aware Neural Machine Translation
[Bastings et al., EMNLP'17]
• Tree-‐‑‒to-‐‑‒Sequence Attentional Neural Machine Translation [Eriguchi et al., ACL'16]
• Towards String-‐‑‒to-‐‑‒Tree Neural Machine Translation [Aharoni and Goldberg, ACL'17]
• Learning to Parse and Translate Improves Neural Machine Translation
[Eriguchi et al., ACL'17]
• Neural Machine Translation with Source-‐‑‒Side Latent Graph Parsing
[Hashimoto and Tsuruoka, EMNLP'17]
24. 1. Introduction
2. Neural Graph-‐‑‒to-‐‑‒Sequence Model
3. Levi Graph Transformation
4. Generation from AMR Graphs
5. Syntax-‐‑‒based Neural Machine Translation
6. Related work
7. Discussion and Conclusion