This document discusses question answering and virtual assistants using deep learning. It provides an overview of question answering systems, including their uses for customer support and knowledge transfer. It describes the typical workflow of initial candidate retrieval using Solr followed by reranking using machine learning models. The document also discusses feature engineering, training data sources, and models for question answering, including supervised models like XGBoost and Siamese neural networks as well as unsupervised models using embeddings. It concludes that deep learning models outperform traditional models with sufficient training data.

1. Question Answering and Virtual
Assistants with Deep Learning
Sanket Shahane
Data Scientist, Lucidworks
Sava Kalbachou
Data Scientist, Lucidworks

2. • http://lucidworks.com/LucidThoughts
• Part 1 of this talk: We talk about the
general business cases and
advancements in Question Answering.
• Part 2 of this talk: We talk about
supervised and unsupervised learning,
Deep Learning.
• If you need a bit more of an
introduction to these concepts check
out Lucid Thoughts!
Make AI Simple

3. • Going Beyond Document Retrieval
• Isn’t it great to have someone answer
your question directly?
• In this fast moving world who wants to read 10
paragraphs when you are looking for just 1
sentence or word?
• Good News! We can do it...
Overview of QA

4. Advancements in Deep Learning
• Recent advances in GPU’s,
and Deep Learning allow us
the possibility to achieve
higher accuracy.
• For example: NER, POS etc,
image recognition, learning
deep representation of the text
• Transfer-learning allows us to
benefit from the work of others.

5. Overview of Question Answering
• 3 types of QA systems

6. Overview of QA
Customer Facing
1.Implement on the help/contact us page
2.Find direct answers
3.Find similarly asked questions
4.Help frame a question
Augmented Intelligence to Support Teams
1.Knowledge transfers
2.Faster Onboarding
3.Faster responses
Data Required:
1.Historic support tickets, FAQ’s, Engagement Logs etc.

7. Overview of QA
Before we dive deep let’s see some examples
http://34.213.226.116:5000/static/question_answering.html

8. 1.Initial candidates retrieving via Solr
• Fast and scalable
• Search among all documents
• Allows to achieve high Recall
2.Reranking using LTR model
• Both classical ML and Deep Learning models might be used
• Reranking is applied to the top N (typically 100~1000) candidates
returned by Solr
• Allows to achieve high Precision
Two steps workflow

9. Feature Engineering - Index Pipeline
We need to use multiple features for ML
models at query time.
Hence, it’s best to pre-compute and index
some expensive ones.

10. Feature Engineering - Query Pipeline
• Synonyms and Answer Type classifier can enrich
your query
• Unsupervised algorithm captures keywords
association between questions and answers
• Q words: indexing, type-ahead, search
• A words: Solr, Lucene

11. Feature Engineering - Query Pipeline
• Named Entities like persons, organizations etc.
• Part of Speech tags like Nouns, Verbs etc.

12. Feature Engineering - Query Pipeline
• Vector representation for the query

13. Feature Engineering - Query Pipeline
• Retrieve pool of candidates from Solr
• Generate LTR features using LTR API
• cosine dist between nouns, verbs
• number of overlapping tokens

14. Feature Engineering - Custom
• Generate additional custom features
• Number of ? marks
• Number of URL’s
• etc.

15. Reranking
• Reranking results by using ML/DL model

16. Fusion UI for Question Answering

17. Fusion UI for Question Answering

18. Fusion UI for Question Answering

19. Deep Dive into QA Modelling
• Training Data: some historic QA transactions
• Support team archives, email threads
• Existing FAQs, forums like StackOverflow
• Slack
• R&D Training Data: IncuranceQA
• Real user questions: 12889
• High quality answers from professionals: 21325
• No labelled data for training?
• Unsupervised methods
• Best working with Question-Question similarity

20. Deep Dive into QA Modelling
• Supervised models:
• XGBoost using heavy feature engineering
• Triplet-based Siamese Deep Neural Networks (DNN)
• Unsupervised models:
• Solr, TFIDF scores including ngrams
• Doc2Vec, Word2Vec algorithms
• GloVe and FastText pre-trained word embeddings
• Google Universal Sentence Encoder (GSE)

21. Deep Dive into QA Modelling

22. Deep Dive into QA Modelling

23. Deep Dive into QA Modelling

24. Deep Dive into QA Modelling

25. Deep Dive into QA Modelling

26. Conclusions
• DL models highly outperform traditional models when
reasonable amount of training data is used
• When training dataset is small (~3-5K QA pairs) their
performance is very similar to traditional models
• There is a significant advantage of using supervised methods
over unsupervised
• Although, unsupervised methods might be used as additional
features or for initial modelling (especially GSE and for QQ)

27. Learn More:
lucidworks.com/LucidThoughts
https://lucidworks.com/contact/

28. Questions?

29. Thank you