This document describes research on using convolutional neural networks (CNNs) for visual sentiment prediction. It presents the methodology which includes fine-tuning a pre-trained CNN model on two datasets labeled for sentiment. Experiments are described that analyze the CNN layer-by-layer, ablate layers, and add new layers. The results show that fine-tuning achieves good performance and deeper layers learn sentiment-related concepts. Future work aims to expand the datasets and analyze network receptive fields.

1. LAYER-WISE CNN SURGERY
FOR VISUAL SENTIMENT
PREDICTION
Víctor Campos Xavier Giró Amaia Salvador Brendan Jou
July 20th 2015

2. Outline
1. Introduction
2. Related work
3. Methodology and results
4. Conclusions
5. Future work
2

3. 3
Introduction: motivation

4. 4
Introduction: motivation

5. Introduction: motivation
5

6. 6
Introduction: problem definition
▷ What?
▷ How?

7. ▷ What? Predict the sentiment that an image provokes to a human
▷ How?
7
Introduction: problem definition

8. ▷ What? Predict the sentiment that an image provokes to a human
▷ How?
8
Introduction: problem definition

9. ▷ What? Predict the sentiment that an image provokes to a human
▷ How? Using Convolutional Neural Networks (CNNs)
9
CNN
Introduction: problem definition

10. 10
CNN
Introduction: example

11. 11
CNN
Introduction: example

12. Outline
1. Introduction
2. Related work
3. Methodology and results
4. Conclusions
5. Future work
12

13. Related work: low-level descriptors
13
Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010, October).
Analyzing and predicting sentiment of images on the social web. In
Proceedings of the international conference on Multimedia (pp. 715-718).
ACM.
Machajdik, J., & Hanbury, A. (2010, October). Affective image
classification using features inspired by psychology and art theory. In
Proceedings of the international conference on Multimedia (pp. 83-92).
ACM.

14. 14
Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S. F. (2013, October). Large-scale visual sentiment ontology and detectors using
adjective noun pairs. In Proceedings of the 21st ACM international conference on Multimedia (pp. 223-232). ACM.
Related work: SentiBank

15. Related work: CNNs for sentiment prediction
15
You, Q., Luo, J., Jin, H., & Yang, J. (2015). Robust image sentiment analysis using progressively trained and domain transferred
deep networks. In The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI).

16. Outline
1. Introduction
2. Related work
3. Methodology and results
a. Convolutional Neural Networks
b. Datasets
c. Experimental setup and results
4. Conclusions
5. Future work
16

17. Convolutional Neural Networks
17
Krizhevsky, A.; Sutskever, I. & Hinton, G. E.: ImageNet Classification with Deep Convolutional Neural Networks. In: NIPS., 2012

18. Outline
1. Introduction
2. Related work
3. Methodology and results
a. Convolutional Neural Networks
b. Datasets
c. Experimental setup and results
4. Conclusions
5. Future work
18

19. Datasets
19
Flickr Twitter
Authors Borth et al. (2013) You et al. (2015)
Size ~500k 1269
Annotation method Textual tags
5 human
annotators

20. Datasets
20
Size
Flickr
dataset
Quality of the
annotations
Twitter
dataset

21. Datasets
21
Size
Flickr
dataset
Quality of the
annotations
Twitter
dataset

22. Outline
1. Introduction
2. Related work
3. Methodology and results
a. Convolutional Neural Networks
b. Datasets
c. Experimental setup and results
4. Conclusions
5. Future work
22

23. Experimental setup: 5-fold cross-validation
Dataset

24. Experimental setup: 5-fold cross-validation

25. Train Test
Experimental setup: 5-fold cross-validation

26. Train Test
Mean ± Std. Dev.
Experimental setup: 5-fold cross-validation

27. 27
ARCHITECTURE
CaffeNet
Experimental setup: CNN

28. 28
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
Experimental setup: CNN

29. Experimental setup: CNN
29
Pre-trained
Model
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]

30. Experimental setup: outline
1. Fine-tuning CaffeNet
2. Layer by layer analysis
3. Layer ablation
4. Layer addition
30

31. Fine-tuning CaffeNet
31

32. Fine-tuning CaffeNet
32

33. Fine-tuning CaffeNet
33

34. Fine-tuning CaffeNet
34
Pre-trained
model

35. Data augmentation (oversampling)
35
CNN

36. Data augmentation (oversampling)
36
CNN

37. Data augmentation (oversampling)
37
CNN

38. Data augmentation (oversampling)
38
CNN

39. Data augmentation (oversampling)
39
CNN

40. Data augmentation (oversampling)
40
CNN

41. Data augmentation (oversampling)
41
CNN

42. Fine-tuning CaffeNet
42

43. Experimental setup: outline
1. Fine-tuning CaffeNet
2. Layer by layer analysis
3. Layer ablation
4. Layer addition
43

44. Layer by layer analysis
44

45. Layer by layer analysis
45

46. Experimental setup: outline
1. Fine-tuning CaffeNet
2. Layer by layer analysis
3. Layer ablation
4. Layer addition
46

47. Layer ablation
47
Raw ablation
2-neuron on top

48. Layer ablation
48

49. Layer ablation
49

50. Layer ablation
50
~16M
params
(~25%)

51. Experimental setup: outline
1. Fine-tuning CaffeNet
2. Layer by layer analysis
3. Layer ablation
4. Layer addition
51

52. Layer addition
52

53. Layer addition
53

54. Outline
1. Introduction
2. Related work
3. Methodology and results
4. Conclusions
5. Future work
54

55. Conclusions
55
Pre-trained
model

56. 56
CNN
Conclusions

57. Conclusions
57

58. Outline
1. Introduction
2. Related work
3. Methodology and results
4. Conclusions
5. Future work
58

59. Future work
59
Size
Flickr
dataset
Quality of the
annotations
Twitter
dataset

60. Future work
60
Size
Flickr
dataset
Quality of the
annotations
Twitter
dataset
New
Flickr
dataset

61. Experimental setup: introduction
61
Model
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
DATASET
[Jou’15]

62. 62

63. Acknowledgements
63
Financial supportTechnical support
Albert Gil Josep Pujal

66. Evaluation metric: accuracy

67. Top-5 scores

68. Receptive fields visualization
CONV5, unit 49:
CONV5, unit 51: