Natural Language Processing Advancements By Deep Learning: A Survey

1. Natural Language Processing
Advancements By Deep
Learning: A Survey
Rimzim Thube

2. Introduction
• Natural Language Processing (NLP) is a discipline of computer science
involving natural languages and computers
• NLP helps machines to understand and analyze human language
• Easy due to conduct NLP due to
- advanced computational power
- large datasets
- deep learning

3. Deep Learning Architectures
1. Multi Layer Perceptron
2. Convolutional Neural Networks
3. Recurrent Neural Network
4. Auto-encoders
5. Generative Adversarial Networks

4. Multi Layer Perceptron (MLP)
• MLP has at least three layers — input, hidden, and output layers
• Layer transfers information from the previous layer to the next layer
using neurons
• Neurons do not communicate with each other
• Uses nonlinear activation functions
• Every node forms fully connected network
• Simplest form of Feed-Forward Neural Networks (FNNs)

5. Convolutional Neural Networks (CNN)
• Uses convolution as their mathematical function
• Used to represent data in 2D or 3D map
• Data points represent information correlation
• In image CNNs, the image pixels of the data map are highly correlated
to their neighboring pixels
• After feature extraction, the key image content is captured and
represented in output
• In NLP, the inputs are sentences or documents represented as
matrices
• Used in Sentiment Analysis and Topic Categorisation

6. Recurrent Neural Network (RNN)
• When we feed the output of each FNN as an input to the next one, a
recurrent neural network (RNN) will be constructed
• Sequences of input vectors are fed as the input, in discrete time
frames, one vector at a time
• At each time step, we use parameters of the current hidden layer as
input to the next time step and make predictions
• Hidden layers carry information from the past
• Long Short Term Memory Network (LSTM) widely used RNN

7. Auto-encoders
• Implement unsupervised methods in deep learning
• Consist of sequence to sequence modelling
• Used in dimensionality reduction
• Aim to learn a code representation for each input since there is no
label corresponding to each input
• Decoder constructs an output based on the encoded input.

8. Generative Adversarial Networks (GAN)
• Includes discriminator and a generator
• Is iterative in nature
• Generator network generates a fake sample
• Discriminator network tries to decide whether this sample is real or
fake
• Goal of the generator is to fool the discriminator till it believes the
images are real
• Once the training phase is finished, there is no need for the
discrimination network
• In NLP, GAN is used for text generation

9. One-Hot Representation
• Each unique element that needs to be represented has its dimension
• Results in a very high dimensional and very sparse representation
• No connection between different words in feature space

10. Continuous Bag of Words (CBOW)
• Tries to predict a word given its surrounding context
• Not dependent on the sequential order of words
• Not dependent on probabilistic characteristics
• Mainly used as pre-trained model
• TF-IDF scores is an example of CBOW

11. Word-Level Embedding
• Words with related semantics become highly correlated in the
representation space
• High generalization power
• Learning a distributed representation has advantage of word usage in
context
• Provides similar representations for semantically correlated words

12. Character-Level Embedding
• Can use small model sizes
• Represent words with lower-level language elements
• Uses out of-vocabulary word (OOV) issue occurs when there is no
equivalent vector in the word embedding

13. Seq2Seq Framework
• Input and the output is represented as a sequence
• Used in applications like machine translation, text summarization ,
speech-to-text, and text-to speech
• Comprises of an encoder and a decoder
• Implemented using RNN or LSTM
• Decoder could take advantage of more information such as a context
vector or intra-attention vectors to generate better outputs
• In NLP applications, the output can be improved by using beam
search to find a reasonably good output sequence

14. Reinforcement Learning in NLP
• During the training of the model, the decoder utilizes two inputs, the
previous decoder output state and the real input, to determine its
current output state
• During testing , the decoder fully relies on the previously created
token from the model distribution
• To solve this, remove the ground-truth dependency in training by
solely relying on model distribution to minimise the cross-entropy
loss
• To solve the inconsistency between the training objective and the test
evaluation metric, reinforcement learning can be

15. Datasets
• Benchmarking is the assessment of methods and algorithms to test
their capability to learn specific patterns
• Types of benchmark datasets
- real-world data
- synthetic data which is artificially generated, when the data required
is more than available data or data privacy is a concern
- toy datasets used for demo and visualisation purposes

16. Datasets
• Data should have 3 properties —
- the right data to train on
- sufficient for the evaluation
- accurate to work on
• Creating proper datasets is complicated and of great importance

17. Part-Of-Speech Tagging
• Process of labelling words with their part of speech categories
• WSJ corpus7 dataset used as a benchmark dataset tagging systems
• LSTM networks, bidirectional LSTM networks, LSTM networks with a
CRF8 layer used for sequence tagging
• Sequence tagging includes part of speech tagging, chunking, and
named entity recognition

18. Parsing
• Parsing is assigning a structure to a recognised string
• Constituency Parsing refers in particular to assigning a syntactic
structure to a sentence
• Dependency Parsing shows the structural relationships between the
words in a targeted sentence
• Bidirectional-LSTMs have been used in dependency parsers for
feature representation
• Stack LSTM and has been used in transition-based parsing

19. Semantic Role Labelling
• Process of identification and classification of text arguments
• Used to determine “who” did “what” to “whom” as well as “how,”
“where,” and “when”
• Goal is to extract the semantic relations between the predicate and
the related arguments
• Current state-of-the-art methods employ joint prediction of
predicates and arguments, novel word representation approaches,
and self-attention models

20. Text Classification
• Goal is to assign predefined categories to text parts for preliminary
classification purposes
• Eg. Categorisation of given documents as to political or non-political news
articles
• Use of CNNs for sentence classification in which training the model on top
of pre-trained word-vectors through fine-tuning, has resulted in
considerable improvements in learning task-specific vectors
• LSTM-RNN architecture has been utilised in for sentence embedding with
in web search task
• A RNN in addition to max-pooling with an effective word representation
method is superiority compared to simple window-based neural network
approaches

21. Information Extraction
• identifies structured information from “unstructured” data such as social
media posts and online news
• Subtasks are Named Entity Recognition, Relation Extraction, Coreference
Resolution, and Event Extraction
• Named Entity Recognition - aims to locate and categorize named entities in
context into pre-defined categories such as the names of people and places
• Relation Extraction - aims to find the semantic relationships between entity
pairs
• Coreference Resolution - identification of the mentions in a context that
refer to the same entity
• Event Extraction - recognising trigger words related to an event

22. Sentiment analysis
• Analyse human opinion, sentiments, and even emotions regarding
products, problems, and varied subjects
• Document-level Sentiment Analysis - determine whether the whole
document reflects a positive or negative sentiment. Gated RNN used.
• Sentence-level Sentiment Analysis - opinion expressed in a sentence.
Recursive auto-encoders.
• Aspect-level Sentiment Analysis - targets an opinion, with the
assumption of the existence of the sentiment and its target.
Attention-based LSTMs

23. Machine Translation (MT)
• Neural Machine Translation
- based on an end-to-end neural network
- no need for extensive preprocessing and word alignments
- once it reaches an end-of-sentence (EOS) token, it starts generating
the output sequence
- no need to specify the length of the sequence

24. Question Answering (QA)
• fine-grained version of Information Retrieval (IR)
• Rule-based Question Answering - consists of (1) question read-in, (2)
dictionary lookup for words in the question, (3) syntactic (POS)
analysis of the words in question, (4) content analysis for extracting
the input question, and (5) estimating relevance regarding answering
the input question
• Question answering in the era of deep learning - core algorithm is
based on deep learning. Convolutional neural networks in order to
encode Question-Answer sentence pairs in the form of fixed length
vectors

25. Question Answering (QA)
• Visual Question Answering - Given an input image, Visual Question
Answering (VQA) tries to answer a natural language question about
the image

26. Document Summarisation
• Extractive Summarisation
- goal is to identify the most salient sentences in the document and
return them as the summary.
- prone to generate long and sometimes overlapping summary
sentences
• Abstractive Summarisation
- goal is to generate summary sentences from scratch
- may contain novel words that do not appear in the original document
- Short summary

27. Dialogue Systems
• Task-based Systems: The structure of a task-based dialogue system
usually consists of the following elements:
- Natural Language Understanding (NLU): deals with understanding and
interpreting user’s spoken context by assigning a constituent structure -
- Dialogue Manager (DM): investigates the context and returns a
reasonable semantic-related response.
- Natural Language Generation (NLG): produces an utterance based on
the response provided by the DM component.

28. Non-task-based Systems
• empower a machine with the ability to have a natural conversation
with humans
• chatbots are of one of the following types: retrieval based methods
and generative methods
• Retrieval-based models have access to information resources and can
provide more concise, fluent, and accurate responses
• Generative models have the advantage of being able to produce
suitable responses when such responses are not in the corpus
• An LSTM-based model has been proposed for context and response
vectors creation

29. Conclusion
• Presented a comprehensive survey of the most distinguished works in
Natural Language Processing using deep learning
• Categorised context for introducing different NLP core concepts,
aspects, and applications
• Emphasised the most significant conducted research efforts in each
associated category
• Deep learning and NLP are two of the most rapidly developing
research topics nowadays
• Due to this rapid progress, it is hoped that soon, new effective models
will supersede the current state-of-the-art approaches

30. References
• This presentation is overview of the below IEEE paper —
• https://arxiv.org/pdf/2003.01200.pdf