1. 1
20 2018/12/21

2. n
• ~2013.3, PhD@ ,
• 2013.4~2016.3, @IBM
• 2016.4~2017.8, @ ERATO, NII
• 2017.9~, @ ,
n
•
ECML’11
AISTATS’15,17
•
AISTATS’18
AAAI’17,18
ongoing
2

3. •
•
•
• AISTATS’18
• AAAI’17,18
3

4. AI
n
•
n
•
4
…

5. AI
n
•
n
•
5
XX
XX

6. AI
n
• AI
AI
AI AI
n
• AI
•
n
6

7. n
•
•
• AI
•
•
7

8. 8

9. EU GDPR
n GDPR-22
1. The data subject shall have the right not to be subject to a decision based
solely on automated processing, including profiling, which produces legal
effects concerning him or her or similarly significantly affects him or her.
2. Paragraph 1 shall not apply if the decision: is necessary for entering into, or
performance of, a contract between the data subject and a data controller; is
authorised by Union or Member State law to which the controller is subject
and which also lays down suitable measures to safeguard the data subject’s
rights and freedoms and legitimate interests; or is based on the data subject’s
explicit consent.
3. In the cases referred to in points (a) and (c) of paragraph 2, the data controller
shall implement suitable measures to safeguard the data subject’s rights and
freedoms and legitimate interests, at least the right to obtain human
intervention on the part of the controller, to express his or her point of view
and to contest the decision.
4. Decisions referred to in paragraph 2 shall not be based on special categories of
personal data referred to in Article 9(2)1), unless point (a) or (g) of Article 9(2)
applies and suitable measures to safeguard the data subject’s rights and
freedoms and legitimate interests are in place.
9

10. n 2016
• ICML, NIPS
n
• , ,
Vol.33, No.3, pages 366--369, 2018.
• , Qiita
10

11. n AI
11
Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)
https://ieeexplore.ieee.org/document/8466590/
7 SCOPUS,
IEEExplore, ACM Digital Library, Google
Scholar, Citeseer Library, ScienceDirect,
arXiv
“intelligible”,
“interpretable”, “transparency”, “black box”,
“understandable”, “comprehensible”,
“explainable” AI
“Artificial Intelligence”, “Intelligent
system”, “Machine learning”, “deep learning”,
“classifier” , “decision tree”

12. •
•
•
• AISTATS’18
• AAAI’17,18
12

13. n
•
n
• 1.
• 2.
• 3.
• 4.
• …
13

14. n
n /
•
n
•
•
n “ ”
•
14

15. 1.
•
2.
•
3.
•
15

16. n
n Why Should I Trust You?: Explaining the Predictions of
Any Classifier, KDD'16 [Python LIME; R LIME]
n A Unified Approach to Interpreting Model Predictions,
NIPS'17 [Python SHAP]
n Anchors: High-Precision Model-Agnostic Explanations,
AAAI'18 [Python Anchor]
n Understanding Black-box Predictions via Influence
Functions, ICML’17 [Python influence-release]
16

17. n
n Born Again Trees
n Making Tree Ensembles Interpretable: A Bayesian Model
Selection Approach, AISTATS'18 [Python defragTrees]
17

18. n
n [Python+Tensorflow
saliency; DeepExplain]
• Striving for Simplicity: The All Convolutional Net
(GuidedBackprop)
• On Pixel-Wise Explanations for Non-Linear Classifier Decisions
by Layer-Wise Relevance Propagation (Epsilon-LRP)
• Axiomatic Attribution for Deep Networks (IntegratedGrad)
• SmoothGrad: Removing Noise by Adding Noise (SmoothGrad)
• Learning Important Features Through Propagating Activation
Differences (DeepLIFT)
18

19. 19

20. n
n Why Should I Trust You?: Explaining the Predictions of
Any Classifier, KDD'16 [Python LIME; R LIME]
n A Unified Approach to Interpreting Model Predictions,
NIPS'17 [Python SHAP]
n Anchors: High-Precision Model-Agnostic Explanations,
AAAI'18 [Python Anchor]
n Understanding Black-box Predictions via Influence
Functions, ICML’17 [Python influence-release]
20

21. n
n
• LIME, SHAP, Anchor
n
• influence
21

22. LIME
n Why Should I Trust You?: Explaining the Predictions of
Any Classifier, KDD'16 [Python LIME; R LIME]
•
•
22

23. LIME
23

24. LIME
n 2
• One-Hot
•
n LIME
• Adult One-Hot
24

25. LIME
n ! !′
• LIME !′ !# ∈ {0, 1}* +
!,
#
+
n -(!) !0
• - ! ≈ 2 !# ≔ 40 + 46!# for !# ∈ NeighborOf(!0
#
)
• 4 4,
25

26. LIME
n !(#) #%
• ! # ≈ ' #( ≔ *% + *,#( for #( ∈ NeighborOf(#%
(
)
n *
• min
:
∑ <,<> ∈? @AB
C ! C − ' C( E
s.t. * % ≤ G
* %
@AB
C #%
H #%
26

27. LIME
n
n
• vs
•
→ LIME
27

28. SHAP
n A Unified Approach to Interpreting Model Predictions,
NIPS'17 [Python SHAP]
•
• LIME
28

29. SHAP
n 1
n 2
29

30. SHAP
n SHAP LIME
• Adult One-Hot
n . “ ”
SHAP “ ”
30

31. SHAP
n SHAP = LIME
• ! "# = %& + %("# = %& + ∑* %*"*
#
3
n 1. + " = ! "#
•
31

32. SHAP
n SHAP = LIME
• ! "# = %& + %("# = %& + ∑* %*"*
#
3
n 2. "*
#
= 0 ⇒ %* = 0
•
32

33. SHAP
n SHAP = LIME
• ! "# = %& + %("# = %& + ∑* %*"*
#
3
n 3. + " = +, "# + +′
+,
# "# − +,
# "# ∖ 0 ≥ +, "# − +, "# ∖ 0 ⇒ %* +# ≥ %* +
• + +′ "*
#
+′ +
+# "*
#
+
33

34. SHAP
n SHAP = LIME
• ! "# = %& + %("# = %& + ∑* %*"*
#
SHAP
n 3
• %* = ∑+⊆-.
+ ! 01 + 12 !
0!
(4- 5 − 4-(5 ∖ 8))
• Shapley Value
34

35. SHAP
n LIME
• min
$
∑&∈( ) * +, * − . *
/
• ) * =
123
1 456678 & & (1 2|&|)
n
• Linear SHAP:
• Tree SHAP:
• Deep SHAP: DeepLIFT
35

36. Anchor
n Anchors: High-Precision Model-Agnostic Explanations,
AAAI'18 [Python Anchor]
•
36

37. Anchor
37

38. Anchor
n Anchor LIME
• Adult One-Hot
n
• Anchor +
38

39. Anchor
n “ +
•
39

40. Anchor
n !
• " ! ! " = 1
n %(⋅ |!) !
n ! =Anchor
• Anchor ! * " +
,- * ! 1. / 0. 1 ≥ +, ! " = 1
Anchor ! "
40

41. Anchor
n !" # $ 1& ' (& ) ≥ +
• 1 − -
Pr !" # $ 1& ' (& ) ≥ + ≥ 1 − -
41

42. Anchor
n ! =
!
• !
• max
%
&' ( [!(+)] s.t. Pr &' + ! 11 ( 21 3 ≥ 5 ≥ 1 − 7
n 1.
n 2.
• Pr &' + ! 11 ( 21 3 ≥ 5 1 − 7 !
42

43. n
n
• LIME, SHAP, Anchor
n
• influence
43

44. influence
n Understanding Black-box Predictions via Influence
Functions, ICML’17 [Python influence-release]
n ("′, %′)
"
44

45. influence
n ("′, %′)
"
n % = (("; *+), *+
*+ = argmin
2∈4
5
67(8,9)∈:
;(<; +)
*+=6> = argmin
2∈4
5
6∈: ?@A 6B6>
;(<; +)
n <> = ("′, %′) *+
• C+=6> − C+
*+=6> − *+ <>
45
<>
= ("′, %′)

46. influence
n !" = (%′, (′) *+
• ,+-." − ,+
*+-." − *+ !"
n
• !" = (%′, (′) ,+-."
•
n influence
• ,+-." − ,+
,+-." − ,+ ≈ −
1
2
345
-6
78(!"; ,+)
46

47. influence
n Data Poisoning
•
•
47

48. influence
n
•
48
This looks like that: deep learning for interpretable
image recognition, arxiv: 1806.10574.

49. n
n
• LIME
• SHAP
• Anchor
n
• influence
49

50. n
• Interpretable Predictions of Tree-based Ensembles
via Actionable Feature Tweaking, KDD’17
•
50
30 200
20 200
30 300
⭕
⭕
…
…

51. n
• Generating Visual Explanations, ECCV‘16
•
51

52. 52

53. n
n Born Again Trees
n Making Tree Ensembles Interpretable: A Bayesian Model
Selection Approach, AISTATS'18 [Python defragTrees]
53

54. BATrees
n Born Again Trees
•
•
n
• !
!
•
54

55. n
• BATrees
• defragTrees RandomForest
55

56. 56

57. n
n [Python+Tensorflow
saliency; DeepExplain]
• Striving for Simplicity: The All Convolutional Net
(GuidedBackprop)
• On Pixel-Wise Explanations for Non-Linear Classifier Decisions
by Layer-Wise Relevance Propagation (Epsilon-LRP)
• Axiomatic Attribution for Deep Networks (IntegratedGrad)
• SmoothGrad: Removing Noise by Adding Noise (SmoothGrad)
• Learning Important Features Through Propagating Activation
Differences (DeepLIFT)
57

58. n
•
58
DNN

59. n
•
59
DNN

60. n
•
60
DNN

61. n DNN
61

62. n
•
→
•
→
62

63. n ! = # $
n $
n [Simonyan et al., arXiv’14]
$%
&' (
&()
•
→ →
*+ ,
*,-
→
•
→ →
&' (
&()
→
63

64. n [Simonyan et al., arXiv’14]
!"
#$ %
#%&
n
• GuidedBP [Springenberg et al., arXiv’14]
back propagation
• LRP [Bach et al., PloS ONE’15]
• IntegratedGrad [Sundararajan et al., arXiv’17]
• SmoothGrad [Smilkov et al., arXiv’17]
• DeepLIFT [Shrikumar et al., ICML’17]
64

65. n
n
[Python+Tensorflow saliency; DeepExplain]
• Striving for Simplicity: The All Convolutional Net
(GuidedBackprop)
• On Pixel-Wise Explanations for Non-Linear Classifier Decisions
by Layer-Wise Relevance Propagation (Epsilon-LRP)
• Axiomatic Attribution for Deep Networks (IntegratedGrad)
• SmoothGrad: Removing Noise by Adding Noise (SmoothGrad)
• Learning Important Features Through Propagating Activation
Differences (DeepLIFT)
65

66. •
•
•
• AISTATS’18
• AAAI’17,18
66

67. n AISTATS’18
•
n AAAI’17,18
•
67

69. defragTrees
n Making Tree Ensembles Interpretable: A Bayesian Model
Selection Approach, AISTATS'18 [Python defragTrees]
•
•
n
69
when
Relationship ≠ Not-in-family, Wife
Capital Gain < 7370
when
Relationship ≠ Not-in-family
Capital Gain >= 7370
when
Relationship ≠ Not-in-family, Unmarried
Capital Gain < 5095
Capital Loss < 2114
when
Relationship = Not-in-family
Country ≠ China, Peru
Capital Gain < 5095
when
Relationship ≠ Not-in-family
Country ≠ China
Capital Gain < 5095
when
Relationship ≠ Not-in-family
Capital Gain >= 7370
…
…

70. n
n
•
70
y = XOR(x1 < 0.5, x2 < 0.5) + ✏

71. n
n
71
2017 The State of Data Science & Machine LearningYour Year on Kaggle: Most Memorable
Community Stats from 2016

72. n
n
72
R

73. n
•
•
n
73

74. defragTrees
n 1.
• !" #, % &)
n 2. !"( #, % ))
• !"( #, % )) ≈ !"(#, %|&) ) ≪ &
!"( #, % ))
n
• )
• Factorized
Asymptotic Bayesian (FAB) Inference
74
&
)

75. n
n
•
•
•
75
D
Synthetic 2 1000 1000
Spambase 57 1000 1000
MiniBooNE 50 5000 5000
Magic 11 5000 5000
Higgs 28 5000 5000
Energy 8 384 384

76. n
76

77. n
77

78. n
•
n
•
•
78

79. n AISTATS’18
•
n AAAI’17,18
•
79

81. n Enumerate Lasso Solu/ons for Feature Selec/on,
AAAI’17 [Python LassoVariants].
n Approximate and Exact Enumera/on of Rule Models,
AAAI'18.

82. → NO!
n
•
82

83. n
• →
•
n
•
83

84. n
•
n
•
→
84

85. n
•
n
•
→
85

86. n
•
n
•
→
86

87. n
•
n
•
87

88. •
•

89. Lasso
Given: !", $" ∈ ℝ'×ℝ ) = 1, 2, … , .
Find: / ∈ ℝ' s.t. !"
0
/ ≈ $ () = 1, 2, … , .)
/
n
•
•
Lasso ℓ5
/∗ = argmin
=
1
2
>/ − $ @ + B / 5
• Lasso /∗ supp(/∗) = {) ∶ /"
∗
≠ 0}

90. Lasso
n
•
→ Lasso
n Lasso
•
→
n

92. Lasso
n ! ⊆ {$%, $', … , $)} Lasso
Lasso ! = min
3
%
'
45 − 7 ' + 9 5 % s.t. supp 5 ⊆ !
Lasso
! Lasso ! <
supp 5 = !
• 9
• 9
• $%, $', $= , $%, $', $> , $%, $>, $? , $%, $' , …
Lasso !

93. Lawler !-best
1. " #
2. $ ∈ #
" $ "& = " ∖ {$}
Lasso("′) #′
(#&, "′)
3.
4.

94. Lawler !-best
1. " #
2. $ ∈ &
' $ '( = ' ∖ {$}
Lasso('′) &′
(&(, '′)
3.
4.
& = 56, 57, 58
' = 56, 57, 59, 58, 5:
&6 = 56, 57, 58
'6 = 56, 57, 59, 58, 5:

95. Lawler !-best
1. " #
2. $ ∈ &
' $ '( = ' ∖ {$}
Lasso("′) #′
(#(, "′)
3.
4.
"5
( = 67, 68, 69, 6:
"7
( = 65, 68, 69, 6:
"8
(
= 65, 67, 68, 6:
"5 = 65, 67, 68, 69, 6:
#5 = 65, 67, 69
"5 = 65, 67, 68, 69, 6:

96. Lawler !-best
1. " #
2. $ ∈ &
" ' "( = " ∖ {'}
-.//0(2′) &′
(&(, 2′)
3.
4.
(#6
(= 78, 79, 7: , "6
()"6
( = 78, 7;, 79, 7:
"8
( = 76, 7;, 79, 7:
";
(
= 76, 78, 7;, 7:
"6 = 76, 78, 7;, 79, 7:
#6 = 76, 78, 79
"6 = 76, 78, 7;, 79, 7:

97. Lawler !-best
1. " #
2. $ ∈ &
" ' "( = " ∖ {'}
-.//0(2′) &′
(&(, 2′)
3.
4.
(#6
(= 78, 79, 7: , "6
()"6
( = 78, 7;, 79, 7:
"8
( = 76, 7;, 79, 7:
";
(
= 76, 78, 7;, 7:
"6 = 76, 78, 7;, 79, 7:
(#8
(= 76, 7;, 79 , "8
()
#6 = 76, 78, 79
"6 = 76, 78, 7;, 79, 7:

98. Lawler !-best
1. " #
2. $ ∈ &
" ' "( = " ∖ {'}
-.//0(2′) &′
(&(, 2′)
3.
4.
(#6
(= 78, 79, 7: , "6
()"6
( = 78, 7;, 79, 7:
"8
( = 76, 7;, 79, 7:
";
(
= 76, 78, 7;, 7:
"6 = 76, 78, 7;, 79, 7:
(#8
(= 76, 7;, 79 , "8
()
(#;
(= 76, 78, 7: , ";
()#6 = 76, 78, 79
"6 = 76, 78, 7;, 79, 7:

99. Lawler !-best
1. " #
2. $ ∈ #
" $ "& = " ∖ {$}
+,--.("′) #′
(#&, "′)
3.
4.
(#3
&= 45, 46, 47 , "3
&)
(#5
&= 43, 48, 46 , "5
&)
(#8
&= 43, 45, 47 , "8
&)
#5 = 45, 46, 47
"5 = 45, 48, 46, 47
#3 = 43, 45, 46
"3 = 43, 45, 48, 46, 47

100. Lawler !-best
1. " #
2. $ ∈ &
' $ '( = ' ∖ {$}
Lasso("′) #′
(#(, "′)
3.
4.
#5 = 65, 67, 68
"5 = 65, 67, 69, 68, 6:
(#7
(= 65, 69, 68 , "7
()
(#9
(= 65, 67, 6: , "9
()
#7 = 67, 68, 6:
"7 = 67, 69, 68, 6:
"8
( = 69, 68, 6:
":
(
= 67, 69, 6:
";
(
= 67, 69, 68
"7 = 67, 69, 68, 6:

101. Lasso % %
n
• %
• % Lasso

103. 1.
n Thaliana gene expression data (Atwell et al. ’10):
• ! ∈ ℝ$%&%'( 2
• ) ∈ ℝ
• 134

104. 2.
n 20 Newsgroups Data (Lang’95); ibm vs mac
• ! ∈ ℝ$$%&' tf-idf
• ( ∈ {ibm, mac} 2
• 1168
→
bios drive ibm
ide drive ibm
dos os, drive ibm
controller drive ibm
quadra, centris 040, clock mac
windows, bios, controller disk, drive ibm
bios, help, controller disk, drive ibm
centris, pc 610 mac

105. n
•
n Lasso
• Lawler !-best
n
•
•

106. •
•
•
• AISTATS’18
• AAAI’17,18
106

107. n
•
•
107

108. n
n Sanity Checks for Saliency Maps, NeurIPS’18.
•
• Sanity Check
108
“ ”
Guided-BP

109. n
•
•
•
n
• Please Stop Explaining Black Box Models for High-Stakes
Decisions, arXiv:1811.10154
109

110. n
•
•
n
•
110