Invited Keynote by Professor Mirella Lapata (University of Edinburgh) for ACL 2017. 2 Aug 2017. Vancouver, Canada License: CC BY 4.0

1. Motivation
Case Studies
Translating from Multiple Modalities
to Text and Back
Mirella Lapata
School of Informatics
University of Edinburgh
mlap@inf.ed.ac.uk
1 / 70

2. Motivation
Case Studies
Research Goal
Methodology
Global Internet User Survey
72% of Internet users find it frustrating to get irrelevant
information when web searching!
Source: www.internetsociety.org/survey
2 / 70

3. Motivation
Case Studies
Research Goal
Methodology
What Causes User Frustration?
User
frustration
Multimodal
information needs
Information
overload
Inaccessible
content
text, DB, SM
images, video
songs, code
formulas
too many hits
contradiction
duplication
no structure
children
2nd language
learners
laypersons
disabled users
3 / 70

4. Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
By rendering information more accessible by translat-
ing within the same language and between language
and different modalities.
4 / 70

5. Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
riding a horse
5 / 70

6. Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
riding a horse
define function with argument n
if n is not an integer value,
throw a TypeError exception
5 / 70

7. Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
riding a horse
define function with argument n
if n is not an integer value,
throw a TypeError exception
Suggs rushed for 82 yards and
scored a touchdown.
5 / 70

8. Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
riding a horse
define function with argument n
if n is not an integer value,
throw a TypeError exception
Suggs rushed for 82 yards and
scored a touchdown.
The Port Authority gave per-
mission to exterminate Snowy
Owls at NY City airports.
5 / 70

9. Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
riding a horse
define function with argument n
if n is not an integer value,
throw a TypeError exception
Suggs rushed for 82 yards and
scored a touchdown.
The Port Authority gave per-
mission to exterminate Snowy
Owls at NY City airports.
Which animals eat owls?
5 / 70

10. Motivation
Case Studies
Research Goal
Methodology
Big Data
90% of the data in the
world today has been created
in the last two years alone
(Science Daily, 22 May, 2013).
At least 2.5 quintillion bytes
of data is produced every day!
6 / 70

11. Motivation
Case Studies
Research Goal
Methodology
A Brief History of Neural Networks
Source: http://qingkaikong.blogspot.com/
7 / 70

12. Motivation
Case Studies
Research Goal
Methodology
Encoder-Decoder Modeling Framework
Kalchbrenner and Blunsom (2013); Cho et al. (2014); Sutskever et al. (2014);
Karpathy and Fei-Fei (2015); Vinyals et al. (2015).
Source: https://medium.com/@felixhill/
8 / 70

13. Motivation
Case Studies
Research Goal
Methodology
Encoder-Decoder Modeling Framework
1 End-to-end training All parameters are simultaneously
optimized to minimize a loss function on the network’s
output.
2 Distributed representations share strength Better
exploitation of word and phrase similarities.
3 Better exploitation of context We can use a much bigger
context – both source and partial target text – to translate
more accurately.
9 / 70

14. Motivation
Case Studies
Research Goal
Methodology
Encoder-Decoder Modeling Framework
1 End-to-end training All parameters are simultaneously
optimized to minimize a loss function on the network’s
output.
2 Distributed representations share strength Better
exploitation of word and phrase similarities.
3 Better exploitation of context We can use a much bigger
context – both source and partial target text – to translate
more accurately.
Essentially a Conditional Recurrent Language Model!
9 / 70

15. Motivation
Case Studies
Research Goal
Methodology
Encoder-Decoder Modeling Framework
The cat sat on the mat
The cat sat on the mat
10 / 70

16. Motivation
Case Studies
Research Goal
Methodology
Encoder-Decoder Modeling Framework
Seven Days is aSeven Days is a Chinese restaurant
Inform(name=Seven Days, food=Chinese)
11 / 70

17. Motivation
Case Studies
Research Goal
Methodology
Encoder-Decoder Modeling Framework
12 / 70

18. Motivation
Case Studies
Research Goal
Methodology
The Scream
13 / 70

19. Motivation
Case Studies
Research Goal
Methodology
The Scream
14 / 70

20. Motivation
Case Studies
Research Goal
Methodology
The Scream
15 / 70

21. Motivation
Case Studies
Research Goal
Methodology
The Scream
16 / 70

22. Motivation
Case Studies
Research Goal
Methodology
The Scream
17 / 70

23. Motivation
Case Studies
Research Goal
Methodology
The Scream
18 / 70

24. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
In the Remainder of the Talk
We will look at the encoder-decoder framework across tasks and
along these dimensions:
Translation
different modalities
same modality
Data
comparable
parallel
Training Size
S
M
L
Model
encoder
decoder
training objective
19 / 70

25. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
The Simplication Task
20 / 70

26. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
The Simplification Task
Goal: to make text easier to read and understand.
Task: involves a broad spectrum of rewrite operations including
deletion, substitution, insertion and reordering.
Source
Previous calculations show that, due to the solar wind
(which drops 30% of the sun’s mass), Earth could escape
to a higher orbit.
Target
Previous calculations show that Earth could escape to a
higher orbit. This is due to the solar wind, which drops 30%
of the sun’s mass.
21 / 70

27. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
The Simplification Task
Goal: to make text easier to read and understand.
Task: involves a broad spectrum of rewrite operations including
deletion, substitution, insertion and reordering.
Source
Previous calculations show that, due to the solar wind
(which drops 30% of the sun’s mass), Earth could escape
to a higher orbit.
Target
Previous calculations show that Earth could escape to a
higher orbit. This is due to the solar wind, which drops 30%
of the sun’s mass.
21 / 70

28. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
The Simplification Task
Goal: to make text easier to read and understand.
Task: involves a broad spectrum of rewrite operations including
deletion, substitution, insertion and reordering.
Source
These alterations are humble, but assist in circumventing
the difficulties of ascertaining the meaning of obfuscated
sentences.
Target
These alterations are simple, but help in getting around the
difficulties of finding the meaning of confusing sentences.
22 / 70

29. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
The Simplification Task
Goal: to make text easier to read and understand.
Task: involves a broad spectrum of rewrite operations including
deletion, substitution, insertion and reordering.
Source
These alterations are humble, but assist in circumventing
the difficulties of ascertaining the meaning of obfuscated
sentences.
Target
These alterations are simple, but help in getting around the
difficulties of finding the meaning of confusing sentences.
22 / 70

30. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
The Simplification Task
Goal: to make text easier to read and understand.
Task: involves a broad spectrum of rewrite operations including
deletion, substitution, insertion and reordering.
Source
These alterations are humble, but assist in circumventing
the difficulties of ascertaining the meaning of obfuscated
sentences.
Target
These alterations are simple, but help in getting around the
difficulties of finding the meaning of confusing sentences.
Constraints: output must be simpler, grammatical, and preserve the
meaning of the input.
22 / 70

31. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
A Tale of Two Presidents
23 / 70

32. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Simplify?
Zhuetal.(2010)
24 / 70

33. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Simplify?
Zhuetal.(2010)
https://en.wikipedia.org/wiki/Vancouver
24 / 70

34. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Simplify?
Zhuetal.(2010)
https://en.wikipedia.org/wiki/Vancouver
https://simple.wikipedia.org/wiki/Vancouver
24 / 70

35. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Simplify?Xuetal.(2015)
25 / 70

36. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Simplify?
Translation
different modalities
same modality
Data
comparable
parallel
Training Size
S
M
L
Model
encoder
decoder
training objective
26 / 70

37. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Simplify?
Translation
different modalities
same modality
Data
comparable
parallel
Training Size
S
M
L
Model
encoder
decoder
training objective
27 / 70

38. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2
28 / 70

39. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
28 / 70

40. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
28 / 70

41. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
Vanilla encoder-decoder model only learns to copy.
28 / 70

42. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
Vanilla encoder-decoder model only learns to copy.
We enforce task-specific constraints via reinforcement learning
(Ranzato et al., 2016; Li et al., 2016; Narashimhan et al., 2016;
Zhang and Lapata, 2017; Williams et al. 2017). 28 / 70

43. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
View model as an agent which reads source X.
Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X).
Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY|
) and receives reward r.
29 / 70

44. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
View model as an agent which reads source X.
Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X).
Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY|
) and receives reward r.
29 / 70

45. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
Get Action Seq. ˆY
View model as an agent which reads source X.
Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X).
Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY|
) and receives reward r.
29 / 70

46. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
Get Action Seq. ˆY
Simplicity
Model
Relevance
Model
Fluency
Model
View model as an agent which reads source X.
Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X).
Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY|
) and receives reward r.
29 / 70

47. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
Get Action Seq. ˆY
Update Agent
Simplicity
Model
Relevance
Model
Fluency
Model
REINFORCE algorithm
View model as an agent which reads source X.
Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X).
Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY|
) and receives reward r.
29 / 70

48. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Deep Reinforcement Learning
X = x1 x2 x3 x4 x5
ˆY = ˆy1 ˆy2 ˆy3
Get Action Seq. ˆY
Update Agent
Simplicity
Model
Relevance
Model
Fluency
Model
REINFORCE algorithm
Simplicity SARI (Xu et al., 2016), arithmetic average of n-gram
precision and recall of addition, copying, and deletion.
Relevance cosine similarity between vectors representing
source X and predicted target ˆY.
Fluency normalized sentence probability assigned by an LSTM
language model trained on simple sentences. 30 / 70

49. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Results: Newsela Dataset
0
0.5
1
1.5
2
2.5
3
3.5
4
2.73
3.06 3.08
3.28 3.38
MeanHumanRatings(Newsela)
Hybrid
EncDec
PBMT-R
DRESS
Reference
Hybrid: Narayan and Garden (2014); EncDec: vanilla encoder-decoder, PBMT-R:
Wubben et al. (2012); DRESS: Deep REinforcement Sentence Simplication 31 / 70

50. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Results: Wikipedia Dataset
0
0.5
1
1.5
2
2.5
3
3.5
4
2.85
3.21
3.34 3.46 3.46
MeanHumanRatings(Wikipedia)
Hybrid
SBMT
PBMT-R
DRESS
Reference
Hybrid: Narayan and Gardent (2014); SBMT: Xu et al. (2016);
PBMT-R: Wubben et al. (2012); DRESS: own model. 32 / 70

51. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
System Output: Obama’s Farewell Speech
My fellow Americans, it has been the honor of my life to serve you. I
won’t stop. In fact, I’ll be right there with you as a citizen for all my
remaining days. But for now, whether you are young or whether you’re
young at heart, I do have one final ask of you as your president.
My fellow Americans, I ’ll be right there with you as a citizen for all my
remaining days, whether you are young or young at heart, I do have
one final ask of you as your president.
33 / 70

52. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
System Output: Obama’s Farewell Speech
My fellow Americans, it has been the honor of my life to serve you. I
won’t stop. In fact, I’ll be right there with you as a citizen for all my
remaining days. But for now, whether you are young or whether you’re
young at heart, I do have one final ask of you as your president.
My fellow Americans, I ’ll be right there with you as a citizen for all my
remaining days, whether you are young or young at heart, I do have
one final ask of you as your president.
I am asking you to hold fast to that faith written into our founding
documents, that idea whispered by slaves and abolitionists, that spirit
sung by immigrants and homesteaders and those who march for justice.
Hold fast to that faith written into our founding documents, that idea
whispered by slaves and abolitionists.
33 / 70

53. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Take-Home Message
Sequence-to-sequence model with task-specific objective.
RL framework could be used for other rewriting tasks.
Training data is not perfect, will never be huge.
Simplifications are decent, system performs well-out of
domain.
34 / 70

54. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Language to Code
35 / 70

55. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Language to Code
Users tell computers what to do using normal language.
Archive missed calls from
Android to Google Drive
Text me the weather
every morning
Every time you are tagged
in a photo on Facebook,
it will be sent to Dropbox
Text me the weather
every morning
Archive missed calls
from Android to Google
Drive
IFTT Dataset: Quirk et al. (2015) 36 / 70

56. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Language to Code
Archive missed calls from
Android to Google DriveArchive missed calls
from Android to Google
Drive
37 / 70

57. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Language to Code
What is the highest mountain in Alaska?
(argmax $0 (and (mountain:t $0) (loc:t $0 alaska:s))
(elevation:i $0))
GEOQUERY:
https://www.cs.utexas.edu/users/ml/nldata/geoquery.html
38 / 70

58. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Language to Code
Dallas to San Francisco leaving after 4
in the afternoon please
(lambda $0 e (and (>(departure time $0) 1600:ti)
(from $0 dallas:ci) (to $0 san francisco:ci)))
ATIS: https://github.com/mesnilgr/is13
39 / 70

59. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Language to Code
Translation
different modalities
same modality
Data
comparable
parallel
Training Size
S
M
L
Model
encoder
decoder
training objective
40 / 70

60. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Language to Code
Translation
different modalities
same modality
Data
comparable
parallel
Training Size
S
M
L
Model
encoder
decoder
training objective
41 / 70

61. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Sequence Model
Dong and Lapata (2016); Jia and Liang (2016).
Sequence Sequence/Tree
LSTM
answer(J,(compa
ny(J,'microsoft'),j
ob(J),not((req_de
g(J,'bscs')))))
Attention Layer
LSTM
what microsoft jobs
do not require a
bscs?
Input LogicalInput Sequence Sequence Logical
Utterance Encoder Decoder Form
Uses minimal domain (and grammar) knowledge
General model, can be used across meaning representations
It is not guaranteed to generate well-formed trees.
42 / 70

62. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Tree Model
(lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci)))
root
lambda $0 e < n >
and < n >
> < n >
departure time $0
1600:ti
< n >
from $0 dallas:ci
43 / 70

63. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Tree Model
(lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci)))
root
lambda $0 e < n >
and < n >
> < n >
departure time $0
1600:ti
< n >
from $0 dallas:ci
43 / 70

64. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Tree Model
(lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci)))
root
lambda $0 e < n >
and < n >
> < n >
departure time $0
1600:ti
< n >
from $0 dallas:ci
43 / 70

65. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Tree Model
(lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci)))
root
lambda $0 e < n >
and < n >
> < n >
departure time $0
1600:ti
< n >
from $0 dallas:ci
43 / 70

66. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Tree Model
(lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci)))
root
lambda $0 e < n >
and < n >
> < n >
departure time $0
1600:ti
< n >
from $0 dallas:ci
43 / 70

67. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Tree Model
(lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci)))
root
lambda $0 e < n >
and < n >
> < n >
departure time $0
1600:ti
< n >
from $0 dallas:ci
43 / 70

68. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Tree Model
and
LSTM
LSTM
LSTM
LSTM
<n> <n> </s>
>
LSTM
LSTM
LSTM
LSTM
<n> 1600:ti </s> from
LSTM
LSTM
LSTM
LSTM
$0 dallas:ci </s>
LSTM
LSTM
departure
_time
$0
LSTM
</s>
LSTM
LSTM
lambda $0 e <n>
LSTM
LSTM
LSTM
</s>LSTM
LSTM
44 / 70

69. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Sequence-to-Tree Modeltrees in decoder
LSTM
LSTM
LSTM
LSTM
lambda $0 e
and
<n>
LSTM
LSTM
LSTM
LSTM
LSTM
LSTM
<n> <n> </s>
LSTM
</s>
from
LSTM
LSTM
LSTM
LSTM
$0 dallas:ci </s>>
LSTM
LSTM
LSTM
LSTM
<n> 1600:ti </s>LSTM
LSTM
departure
_time
$0
LSTM
</s>
ee >>
LS
LLLLLLLSS
LS
<<<<</<n>>
nndddddddddddd
LST
LLLLLLLLSSSSSSSTT
<<n<
LLLLLSSSSSSTT
LST
> <<<<<<<<<<<//s
LLLLLLLSSSSSSTT
LLLLLLLSSSSSSTT
>>> <<<<<nn>>>>>>>>>>>>>>> n><<<<<<<<<<n><<<
>>
LSTM
LLLLLLLLLSSSSSSSSSTTTM
< >> 1111111111166<n>< >
45 / 70

70. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Results: JOBS
70
75
80
85
90
95
100
90.7
87.1
90
JOBS Dataset
Accuracy(%)
DCS
Seq2Seq
Seq2Tree
46 / 70

71. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Results: GEOQUERY
70
75
80
85
90
95
100
88.9
84.6
87.1
GEOQUERY Dataset
Accuracy(%)
TISP
Seq2Seq
Seq2Tree
47 / 70

72. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Results: IFTT
50
60
70
80
90
100
66.5
72.9 74.2
IFTT Dataset
BalancedF1(%)
possclass
Seq2Seq
Seq2Tree
48 / 70

73. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Structure is Coming Back!
Recurrent Neural Network Grammars. Chris Dyer, Adhiguna Kun-
coro, Miguel Ballesteros and Noah A. Smith. NAACL, 2016.
Incorporating Structural alignment Biases into an Attentional
Neural Translation Model. Trevor Cohn, Cong Duy Vu Hoang,
Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza
Haffari. NAACL, 2016.
Encoding Sentences with Graph Convolutional Networks for
Semantic Role Labeling. Diego Marcheggiani, and Ivan Titov.
EMNLP 2017.
Structured Attention Networks. Yoon Kim, Carl Denton, Loung
Hoang, Alexander M. Rush. ICLR 2017.
49 / 70

74. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Take-home Message
Sequence-to-tree model, hierarchical tree decoder.
Training data small but strictly parallel.
Scaling: cross-domain, cross-language training.
Neural nets can generate structure even though they don’t
know what it is.
50 / 70

75. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Movie Summarization
51 / 70

76. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Why Movies?
The litmus test for understanding (from NLP and CV perspective)
Movies are naturally multimodal and not sequential.
Is it possible to encode a movie into a vector?
52 / 70

77. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Why Summarization?
Anyone can do it! Here’s eight-year old Tru, summarizing Zootopia.
53 / 70

78. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Movie Profile Generation
The Silence of the Lambs can be described
as tense, captivating, and suspenseful. The
plot revolves around special agents, mind
games, and a psychopath. The main genres
are thriller and crime. In terms of style, The
Silence of the Lambs stars a strong female
character. In approach, it is serious and re-
alistic. It is located in Maryland and Virginia.
The Silence of the Lambs takes place in the
1990s. It is based on a book. The movie
has received attention for being a modern
classic, an Oscar winner, and a blockbuster.
Note that The Silence of the Lambs involves
brief nudity and sexual content.
54 / 70

79. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Movie Profile Generation
Mood: Suspenseful, Captivating, Tense, Scary
Plot: Serial Killer, Special Agents, Investigation,
Mind Game, Psychopath, Crimes, Deadly,
Law Enforcement, Mind and Soul, Rivalry
Genre: Crime, Thriller
Style: Strong Female Presence
Attitude: Serious, Realistic
Place: Maryland, USA, Virginia
Period: 20th Century, 90s
Based on: Based on Book
Praise: Award Winner, Blockbuster, Critically Ac-
claimed, Oscar Winner, Modern Classic,
Prestigious Awards
Flag: Brief Nudity, Sexual Content, Strong Violent
Content
55 / 70

80. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Movie Profile Generation
Mood: Suspenseful, Captivating, Tense, Scary
Plot: Serial Killer, Special Agents, Investigation,
Mind Game, Psychopath, Crimes, Deadly,
Law Enforcement, Mind and Soul, Rivalry
Genre: Crime, Thriller
Style: Strong Female Presence
Attitude: Serious, Realistic
Place: Maryland, USA, Virginia
Period: 20th Century, 90s
Based on: Based on Book
Praise: Award Winner, Blockbuster, Critically Ac-
claimed, Oscar Winner, Modern Classic,
Prestigious Awards
Flag: Brief Nudity, Sexual Content, Strong Violent
Content
55 / 70

81. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Generate Movie Profiles?
Scripts (1,280), taken from sources available on-line
Drama (667), Thriller (452), Comedy (379), Action (288)
Paired with corresponding movies (video) and closed captions
Paired with Jinni meta data (movie attributes and profiles)
SCRIPTBASE: Gorinski and Lapata (2015)
56 / 70

82. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Generate Movie Profiles?
Translation
different modalities
same modality
Data
comparable
parallel
Training Size
S
M
L
Model
encoder
decoder
training objective
57 / 70

83. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
How to Generate Movie Profiles?
Translation
different modalities
same modality
Data
comparable
parallel
Training Size
S
M
L
Model
encoder
decoder
training objective
58 / 70

84. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Database-to-Sequence Model
LSTM
Attention LayerLSTM
at microsoft jobs
do not require a
The Silence of the Lambs can be described
as tense, captivating, and suspenseful. The
plot revolves around special agents, mind
games, and a psychopath. The main genres
are thriller and crime. In terms of style, The
Silence of the Lambs stars a strong female
character. In approach, it is serious and re-
alistic. It is located in Maryland and Virginia.
The Silence of the Lambs takes place in the
1990s. It is based on a book. The movie
has received attention for being a modern
classic, an Oscar winner, and a blockbuster.
Note that The Silence of the Lambs involves
brief nudity and sexual content.
59 / 70

85. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Database-to-Sequence Model
LSTM
Attention LayerLSTM
at microsoft jobs
do not require a
The Silence of the Lambs can be described
as tense, captivating, and suspenseful. The
plot revolves around special agents, mind
games, and a psychopath. The main genres
are thriller and crime. In terms of style, The
Silence of the Lambs stars a strong female
character. In approach, it is serious and re-
alistic. It is located in Maryland and Virginia.
The Silence of the Lambs takes place in the
1990s. It is based on a book. The movie
has received attention for being a modern
classic, an Oscar winner, and a blockbuster.
Note that The Silence of the Lambs involves
brief nudity and sexual content.
60 / 70

86. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Database-to-Sequence Model
LSTM
Attention Layer
LSTM
at microsoft jobs
o not require a
The Silence of the Lambs can be described
as tense, captivating, and suspenseful. The
plot revolves around special agents, mind
games, and a psychopath. The main genres
are thriller and crime. In terms of style, The
Silence of the Lambs stars a strong female
character. In approach, it is serious and re-
alistic. It is located in Maryland and Virginia.
The Silence of the Lambs takes place in the
1990s. It is based on a book. The movie
has received attention for being a modern
classic, an Oscar winner, and a blockbuster.
Note that The Silence of the Lambs involves
brief nudity and sexual content.
61 / 70

87. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Content Selection
Wen et al. (2015, 2016)
labelset(genre1=’action’,genre2=’sci-fi’,…)
0
...
1
d0
rt
1
0
Ct
ot
it
ft
DA cell
LSTM cell
wt
ht-1
wt
ht-1
wt
ht-1
wt
ht-1
wt
ht-1
dt-1
dt
ht
CS cell
62 / 70

88. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Alignment between Video, Screenplay and CCs
subshot1
DR. LECTER’S CELL is coming slowly INTO
VIEW... Behind its barred front wall
is a second barrier of stout nylon
net... Sparse, bolted- down furniture,
many softcover books and papers. On
the walls, extraordinarily detailed,
skillful drawings, mostly European
cityscapes, in charcoal or crayon.
Clarice stops, at a polite distance
from her bars, clearing her throat.
CLARICE Dr. Lecter... My name is
Clarice Starling. May I talk with you?
DR. LECTER Good morning.
CUTING BETWEEN THEM as Clarice comes a
measured distance closer.
DR. LECTER You’re one of Jack
Crawford’s, aren’t you?
CLARICE I am, yes.
Dr LECTER May I see your credentials?
00:04:38 –> 00:04:39 Dr.
Lecter... My name is Clarice
Starling. May I talk with you?
00:04:40 –> 00:04:41 Good
morning.
00:04:45 –> 00:04:46 You’re
one of Jack Crawford’s, aren’t
you?
00:04:47 –> 00:04:48 I am,
yes.
00:04:49 –> 00:04:50 May I
see your credentials?
closed captions
closed captions
screenplayvideo
subshot2
63 / 70

89. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Multi-label Assignment Using SPENs
w1
w2
wn i1
i2
im
x .........
F(x)
g1
g2
...g3
gL
EglobalElocal
.... . . .
sum
Structured Prediction Energy
Networks (Belanger and McCallum 2016)
First network generates
f-dimensional feature vector F(x)
for input x.
Second network finds label
assignment g s.t.
min
g
Ex (g) = Elocal
x (g) + E
global
x (g)
Features: words, sentiment,
character graph, interactions,
objects in scene, scene duration,
movement in scene.
64 / 70

90. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Attribute Evaluation
Mood Plot Genre Attitude Place Flag
0
20
40
60
80
100
52.2
38.9
55.7
71.4
50.4
47.6
59.5
44.6
65.1
72
49.6
55.9
AttributeIdentification(%F1)
LibLinear
SPEN
65 / 70

91. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Profile Evaluation
30
40
50
60
70
80
90
100
47.5
55
60
89.17
Percentageofprofilesratedsuitable
kNN
Template
LSTM
Jinni
66 / 70

92. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
System Output
The Bourne identity can be described as suspense-
ful, stylized, and rough. The plot centers around a
deadly, themes of uncover truth, and investigation.
The Bourne identity is an action thriller, and crime
movie. In approach, the Bourne identity is realistic
and serious. The Bourne identity is set, at least in
part, in the Europe. Note that the Bourne identity in-
volves violent content.
67 / 70

93. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
Take-Home Message
Encoder performs content selection over set of
attribute-value pairs.
Neural networks are well suited to tasks involving multiple
modalities.
There is too much data, but not in terms of number of
training examples.
68 / 70

94. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
The Future
The future is uncertain but this uncertainty is at the very
heart of human creativity. Ilya Prigogine
69 / 70

95. Motivation
Case Studies
The Simplication Task
Language to Code
Movie Summarization
A Big Thank You to
Philip Gorinski Xingxing Zhang Li Dong Siva Reddy Jianpeng Cheng
Jonathan Mallinson Yang Liu Stefanos Angelidis Spandana Gella Carina Silberer
Lea Frermann Laura Perez Mark Steedman Shay Cohen Rico Sennrich
70 / 70