2019/09/05
LGCNS AI Tech Talk for NLU (feat.KorQuAD)
- 서울대학교 김대식님
- accepted for 2019 ACL
- Textbook Question Answering (TQA) with Multi-modal Context Graph Understanding and Self-supervised Open-set Comprehension
6. … and the title is …
Q What is the title of this paper?
https://media-assets.bookbub.com/wp-content/uploads/
2015/12/cute-baby-skim-reading-gif.gif
Context
Solving step
8. Q Plates move over Earth’s surface
because of _________
a) conduction within the crust.
b) radiation from the inner core.
c) subduction in the outer core.
d) convection within the mantle.
http://www.grygla.k12.mn.us/uploads/2/3/7/1/23718473/earth_sci_-_ch._6_-_plate_tectonics.pdf
Context
9. Q Plates move over Earth’s surface
because of _________
a) conduction within the crust.
b) radiation from the inner core.
c) subduction in the outer core.
d) convection within the mantle.
Q What is the southern most point
of the ring of fire?
a) South sandwich trench
b) Japan trench
c) Aleutian trench
d) Kurile trench
http://www.grygla.k12.mn.us/uploads/2/3/7/1/23718473/earth_sci_-_ch._6_-_plate_tectonics.pdf
10. (Kembhavi et al., 2017)
single sentence / multiple sentences within a paragraph
question/context diagram, and one paragraph
21. Multi-modal context graph understanding: visual context
rabbit
connects to
fox
mouse
ladybird
snake
14 objects
22 stages
…
diagram parsing from UDPnet
(Kim et al., 2018)
parsed info from UDPnet
& OCR info from TQA
build context graph
of diagrams
rabbit
fox
mouseladybird
14
objects
22
stages
context matrix 𝑪 𝒅
adjacency matrix 𝑨 𝒅
22. Multi-modal context graph understanding: textual context
… and the title is …
Context
Q What is the title of this paper?
anchor node
dependency
parsing
filtering by
anchor
nodes
context matrix 𝑪 𝒕
adjacency matrix 𝑨𝒕
3
1
1
1
1
2
2
26. Experiments: quantitative results
Model Text T/F Text MC Text All Diagram All
Random 50.10 22.88 33.62 24.96 29.08
MemN+VQA (Kembhavi et al., 2017) 50.50 31.05 38.73 31.82 35.11
MemN+DPG (Kembhavi et al., 2017) 50.50 30.98 38.69 32.83 35.62
BiDAF+DPG (Kembhavi et al., 2017) 50.40 30.46 38.33 32.72 35.39
Challenge - - 45.57 35.85 40.48
IGMN (Li et al., 2018) 57.41 40.00 46.88 36.35 41.36
Our full model w/o visual context 62.32 49.15 54.35 36.61 45.06
Our full model w/ f-GCN2 62.22 48.76 54.11 37.72 45.52
Our full model 62.73 49.54 54.75 37.61 45.77
27. Experiments: quantitative results
Model Text T/F Text MC Text All Diagram All
IGMN (Li et al., 2018) 57.41 40.00 46.88 36.35 41.36
Our full model w/o visual context 62.32 49.15 54.35 36.61 45.06
w/o SSOC(VAL) 60.82 49.08 53.72 36.53 44.72
w/o SSOC(TR+VAL) 60.72 46.34 52.02 36.57 43.93
w/o f-GCN & SSOC(TR+VAL) 58.62 44.77 50.24 35.20 42.36
Our full model w/ f-GCN2 62.22 48.76 54.11 37.72 45.52
w/o SSOC(VAL) 62.63 48.43 54.03 37.32 45.28
w/o SSOC(TR+VAL) 61.42 46.67 52.49 36.71 44.22
w/o f-GCN & SSOC(TR+VAL) 58.72 45.16 50.51 35.67 42.74
Our full model 62.73 49.54 54.75 37.61 45.77
w/o SSOC(VAL) 62.22 48.82 54.11 37.47 45.39
w/o SSOC(TR+VAL) 60.02 46.86 52.06 36.61 43.97
w/o f-GCN & SSOC(TR+VAL) 58.72 45.16 50.51 35.67 42.74
SSOC : Self-Supervised Open-set Comprehension
28. Experiments: quantitative results
Model Text Diagram All
Our full model w/o SSOC 52.06 36.61 43.97
w/o anchor flag (q) 49.29 35.78 42.21
w/o anchor flag (a) 43.24 31.50 37.09
w/o anchor flag (q & a) 42.64 31.72 36.92
29. Experiments: qualitative results
… lithosphere and asthenosphere are
layers based on physical properties .
the outermost layer is the lithosphere .
the lithosphere is the crust and the
uppermost mantle . in terms of physical
properties , this layer is rigid , solid ,
and brittle . it is easily cracked or
broken . below the lithosphere is the
asthenosphere . the asthenosphere is
also in the upper mantle . this layer is
solid , but it can flow and bend . a
solid that can flow is like silly putty ..…
Q what layer is directly below the crust ?
a) mantle
b) core
c) inner layer
d) space
asthenosphere
crust
…
…
uppermost
lithosphere
mantle
below
layer
outer
middle
layer
mantle
crust
30. Experiments: qualitative results
… runoff carved channels in the soil in
figure 19.1 . running water causes most
soil erosion , but wind can carry soil
away too . what humans do to soil
makes it more or less likely to be eroded
by wind or water . human actions that
can increase soil erosion are described
below .…
Q the main cause of soil erosion is ____
a) wind .
b) ice wedging .
c) abrasion .
d) running water .
causes
water
running
…
…
erosion
soil
31. Experiments: qualitative results
… the dense , iron core forms the center
of the earth . scientists know that the
core is metal from studying metallic
meteorites and the earths density .
seismic waves show that the outer core
is liquid , while the inner core is solid .
movement within earths outer liquid iron
core creates earths magnetic field . these
convection currents form in the outer
core because the base of the outer core
is heated by the even hotter inner core…
Q convection currents occur in the inner core .
a) true
b) false
form
core
currents
outer convection
these
in
……
32. Conclusion & Further work
Multi-modal understanding for TQA problems
more paragraphs
visual information
Self-supervised open-set comprehension
without Q-A-Context triple
State-of-the-art performance
still a lot of room for improvement