For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/sept-2017-alliance-vitf-samek For more information about embedded vision, please visit: http://www.embedded-vision.com Dr. Wojciech Samek of the Fraunhofer Heinrich Hertz Institute delivers the presentation "Methods for Understanding How Deep Neural Networks Work" at the Embedded Vision Alliance's September 2017 Vision Industry and Technology Forum. In his presentation, Dr. Samek covers the following topics: ▪ Unbeatable AI systems ▪ Deep neural network overview ▪ Opening the "black box" ▪ Summary

1. Methods for Understanding How
Deep Neural Networks Work
Dr. Wojciech Samek
Head of Machine Learning Group
Fraunhofer Heinrich Hertz Institute
Vision Industry and Technology Forum 6th Sep 2017, Hamburg, Germany

2. Unbeatable AI Systems
AlphaGo beats Go
human champ
Deep Net outperforms humans
in image classification
Deep Net beats human at
recognizing traffic signs
DeepStack beats
professional poker players
Computer out-plays
humans in "doom"
Autonomous search-and-rescue
drones outperform humans
IBM's Watson destroys
humans in jeopardy

3. Elon Musk (2017): “AI will be able to beat humans at
EVERYTHING by 2030.”
Unbeatable AI Systems
You may heard such promises before
H. A. Simon (1965): "Machines will be capable, within
twenty years, of doing any work a man can do."
M. Minsky (1970): "In from three to eight years we will
have a machine with the general intelligence of an
average human being."

4. Computing power
Very large deep neural networks Information (implicit)
Solve task
Huge volumes of data
Past promises did not become reality, but today …

5. 650,000 neurons, 60 million parameters
Deep Neural Networks
Why are these models so successful ?
- “end-to-end” training
- feature hierarchy
- distributed / reusable representation

6. Deep Neural Networks: “End-to-End” Training
“Slippery road”
Traditional Approaches
input feature extraction classification output
Deep Learning
“Slippery road”
input feature extraction + classification output

7. (Source: Yann LeCun
Deep Neural Networks: Feature Hierarchy
Low level
features
Mid level
features
High level
features
Classifier
“Car”

8. Hierarchical information processing in the brain
Deep Neural Networks: Feature Hierarchy
(Source: Simon Thorpe)

9. Deep Neural Networks: Reusable Representation
Features extracted from “car” images can be used for
other tasks / classification of other objects.
objects with
roundish shape

10. Very large deep neural networks Information (implicit)
What did the neural network learn ?
How does it solve the problem ?
Can we extract human interpretable information ?
Do we understand the AI ?

11. Can we trust these “end-to-end” trained black box algorithms ?
Black-Box Systems
(More information: https://neil.fraser.name/writing/tank/)
Already in the 1980s neural networks have been used for
classification tasks, e.g., identify tanks in the forest.
Training Dataset:
- 100 photos of tanks hiding behind trees
- 100 photos of trees with no tanks.
First results were promising, but did
network solve the task correctly ?

12. Understand “weaknesses” of classifier
Detect biases / bring in human intuition.
Learn from the learning machine
“I've never seen a human play this move.” (Fan Hui)
Wrong decisions can be harmful and costly.
Verify that system works as expected
Interpretability in the sciences
The “why” often more important than the prediction.
Compliance to legislation
“right to explanation”, retain human decision …
We need to “open” Black-Box Systems

13. “rooster”
We developed a general method to explain
individual classification decisions.
Main idea:
Opening the Black-Box
Layer-wise Relevance Propagation (LRP)
(Bach et al., PLOS ONE, 2015)
“measure how much each pixel
contributes to the overall prediction”

14. Classification
cat
rooster
dog
Opening the Black-Box

15. Explanation
cat
rooster
dog
=
Initialization
Idea: Backpropagate “relevances”
Opening the Black-Box

16. Opening the Black-Box
Explanation
cat
rooster
dog
?
Intuition: Redistribute relevance proportionally
- a neuron gets more relevance if it’s more activated
- more relevance flows over strong connections

17. Opening the Black-Box
Explanation
cat
rooster
dog
Theoretical interpretation: Deep Taylor Decomposition
(Montavon et al., Pattern Recognition, 2017)

18. [number]: explanation target class
red color: evidence for prediction
blue color: evidence against prediction
what speaks for / against
classification as “3”
what speaks for / against
classification as “9”
Opening the Black-Box

19. Opening the Black-Box

20. Application: Compare Classifiers
Two classifiers
- similar classification accuracy on horse class
- but do they solve the problem similarly ?
(Lapuschkin et al., IEEE CVPR, 2016)

21. Application: Compare Classifiers
Images from
PASCAL VOC
2007 dataset
Interpretability helps to
- understand biases / flaws in the data and weaknesses of the classifier
- verify that system works as expected

22. Application: Compare Classifiers
GoogleNet focuses on
faces of animal.
—> suppresses background noise
(Binder et al., ICML Visualization
Workshop, 2016)
Interpretability helps to
- compare and select models / architectures

23. Application: Measure Context Use
classifier
how important
is context ?
how important
is context ?
relevance outside bbox
relevance inside bbox
importance
of context
=
(Lapuschkin et al., IEEE CVPR, 2016)

24. Image Fisher Vector DNN
boat
boat
Contextuse
airplane
airplane
sofachair chair sofa
Application: Measure Context Use
Interpretability helps to
- extract additional classification-related information

25. Application: Video Analysis
Motion vectors can be extracted
from the compressed video
-> allows very efficient analysis

26. Application: Video Analysis
Interpretability helps to
- extract additional classification-related information
- compare features

27. Other Applications
Identifying age-related features
(Arbabzadah et al., GCPR, 2016)
(Sturm et al., J Neuroscience Methods, 2016)
Brain-Computer Interfacing
Identifying relevant words in text
(Arras et al., PLOS ONE, 2017)
Detection of morphing attacks
(Seibold et al., IWDW, 2017)

28. In many problems interpretability as important as prediction
(trusting a black-box system may not be an option).
Use in practice
- verify predictions, detect biases and flaws, debug models
- compare and select architectures, understand and improve models
- extract additional information, perform further tasks
We have a powerful, mathematically well-founded method to explain
individual predictions of complex machine learning models.
More research needed on how to compare and evaluate all the different
aspects of interpretability.
Summary

29. Thank you for your attention
For more information, check out our tutorial paper:
Montavon et al. “Methods for Interpreting and Understanding Deep Neural Networks”
https://arxiv.org/abs/1706.07979