Video: https://www.youtube.com/watch?v=Tgvs9WfA3LI#t=5h8m16s

1. Applying Deep Learning in
Information Retrieval System

2. http://twchrislin.com/

3. My project
http://twchrislin.com/project/
Tools:
python
numpy
pandas
scikit-learn
xgboost
theano
lasagne
id LifeTime dira FlightDistance FlightDistanceError … signal
18453471 0.00157832377590239050.999999344348907614.0333347320556640.6814009547233582 … 0
5364094 0.00098804314620792870.99970477819442765.536156654357910.3023407459259033 … 0
11130990 0.00087667378829792140.99998414516448966.1173019409179690.27646270394325256… 0
15173787 0.0008541746647097170.99990290403366085.2280673980712890.22073912620544436… 0
1102544 0.00112933525815606140.999995291233062939.069534301757811.8981972932815552 … 0
https://youtu.be/_SjgCoR7e1s

4. http://www.emotibot.com/
:
python
numpy
pandas
scikit-learn
xgboost
theano
lasagne
Machine Learning
Deep Learning

5. Outline
1. Information Retrieval
2. Neural Networks and Convolutional Neural Networks
3. Feature Vector Representation
4. Sentence Retrieval
5. Image Retrieval
6. Conclusion

6. Information Retrieval
Input Query
Encoding
Database
Matching
Ranking

7. Neural Networks
http://speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2015_2/Lecture/DNN%20(v4).pdf
Single Neuron, Logistic Regression
Sigmoid funciton
28 pixels
28 pixels
784 pixels

8. Neural Networks
http://speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2015_2/Lecture/DNN%20(v4).pdf
Multiclass 10

9. Neural Networks
http://speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2015_2/Lecture/DNN%20(v4).pdf
MLP Multiclass Z1xL1=X1x785 W785xL1
+
f(Z1xL1)
Sigmoid funciton
Z1xL2=X1xL1 WL1xL2
+
f(Z1xL2)

10. Convolutional Neural Networks
http://www.slideshare.net/tw_dsconf/ss-62245351
translation
invariance
subsampling
Conv + Pooling Conv + Pooling MLP

11. Feature Vector Representation
Bengio, 2014, Representation Learning: A Review and New Perspectives
FeatureVector
Representation
Learning
http://deeplearning4j.org/convolutionalnets
NN
feature
extractor

12. Feature Vector Representation
Input
Neural Networks
including
MLP, CNN, RNN
Feature Vector
[ 0.25, 0.33…, 0.78 ]
Encoding
Matching
Feature Vector 1
Feature Vector 2
Similarity
Function
Score
• cosine similarity
• v1
TM v2

13. Outline
1. Information Retrieval
2. Neural Networks and Convolutional Neural Networks
3. Feature Vector Representation
4. Sentence Retrieval
5. Image Retrieval
6. Conclusion

14. <Pad>
<Pad>
<Pad>
<Pad>
Sentence Retrieval
http://emnlp2014.org/papers/pdf/EMNLP2014181.pdf
Yoon Kim, 2014, Convolutional Neural
Networks for Sentence Classification
Neural Net Architecture
NN
feature
extractor
1 of K (one-hot)
+
word embedding
Conv + MaxPooling

15. Sentence Retrieval
Results:
• Training acc: 99% up
• Validation acc: 99% up
Experiment:
• 5 classes sentences
• Training Set 80%
• Validation Set 20%
Feature vector PCA tSNE

16. Sentence Retrieval
Recurrent Neural Network
Kyunghyun Cho et. al., 2014,
Learning Phrase Representations using
RNN Encoder–Decoder for Statistical
Machine Translation
https://devblogs.nvidia.com/parallelforall/
introduction-neural-machine-translation-gpus-part-2/

17. Skip-Thought Vectors
Skip-Thought Vectors http://arxiv.org/pdf/1506.06726v1.pdf
Encode a sentence into a thought vector hi by predicting its neighbor
sentence.
Ryan Kiros et. al., 2015, Skip-Thought Vectors
… I just got back home. I could see the cat on the steps. This was strange. …

18. Channel A
http://www.cherry123.com/data/book/14299/book14299_2.html
Skip-Thought Vectors
?
?
?
?
?
?
?
! ? !
?
, ,

19. Semantic Similarity
Skip-Thought Vectors
query sen skt result skt score
… … 0.934
0.918
0.901
0.898
0.889
0.861
0.836
0.831
0.812
0.777
0.765

20. Reference Relationship
Skip-Thought Vectors
query sen skt result skt score
0.898
0.916
0.836
… … 0.755
0.757
0.763

21. Limitations
Skip-Thought Vectors
• Requiring huge size of corpus
• … I just got back home. I could see the cat on the steps. This was strange. …
• … I got back to office. I could see the cat on the steps. This was cool. …
11,051 novels with 17,515,150 sentences
• Scenario dependency
query sen skt result skt score
0.697
… … 0.697
0.697

22. Something Interesting
Skip-Thought Vectors
query sen skt result skt score
0.889
0.697
skip-thought knows: =
I’m ?
skip-thought knows: =
?

23. Image Retrieval
“A cat” Image Input
Expected Outcome:
Feature Vector
Calculation
Matching
Database

24. Image Retrieval
Input
Image
pre-
processing
(2)
(1)
Hashing
Vector
Layer
FC + Softmax
Layer
.
.
.
.
.
.
CNN
network
1st: various targets
Step1
Training
copy
CNN
network
Step2
Training
Kevin Lin et. al, 2015, Deep Learning of Binary Hash Codes for Fast Image Retrieval
.
.
.
.
.
.
2nd: task specific targets
encoding
application Hashing
Vector
Layer
copy
CNN
network
copy
Feature Vector
AlexNet
VGG

25. Image Retrieval
Kevin Lin et. al, 2015, Deep Learning of Binary Hash Codes for Fast Image Retrieval
Demo
http://tweddielin.pythonanywhere.com/
https://github.com/tweddielin/flask-imsearch
Github

26. Conclusion
1. Deep Learning => Trainable Feature Extractor
2. Type of Neural Nets: MLP, CNN, RNN
3.
• http://www.slideshare.net/tw_dsconf/ss-62245351
• http://speech.ee.ntu.edu.tw/~tlkagk/courses_MLSD15_2.html

27. Backup

28. Convolutional Neural Networks
http://cs231n.github.io/convolutional-networks/
More Information
1 1 2 4
5 6 7 8
3 2 1 -2
1 2 3 4
ReLU
max(0,x)
Convolved Feature
MaxPooling