"Understanding Black-box Predictions via Influence Functions" (2017), PangWei Koh and Percy Liang, ICML2017 Best paper

1. Terry Taewoong Um (terry.t.um@gmail.com)
University of Waterloo
Department of Electrical & Computer Engineering
Terry T. Um
UNDERSTANDING BLACK-BOX PRED
-ICTION VIA INFLUENCE FUNCTIONS
1

2. TODAY’S PAPER
Terry Taewoong Um (terry.t.um@gmail.com)
ICML2017 best paper
https://youtu.be/0w9fLX_T6tY

3. QUESTIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• How can we explain the predictions of a black-box model?
• Why did the system make this prediction?
• How can we explain where the model came from?
• What would happen if the values of a training point where
slightly changed?

4. INTERPRETATION OF DL RESULTS
Terry Taewoong Um (terry.t.um@gmail.com)
• Retrieving images that maximally activate a neuron [Girshick et al. 2014]
• Finding the most influential part from the image [Zhou et al. 2016]
• Learning a simpler model around a test point [Ribeiro et al. 2016]
But, they assumed a
fixed model
 My NN is a function
of training inputs

5. INFLUENCE OF A TRAINING POINT
Terry Taewoong Um (terry.t.um@gmail.com)
• What is the influence of a training example for
the model (or for the loss of a test example)?
Optimal model param. :
Model param. by training w/o z :
Model param. by upweighting z :
without z == (𝜖 = −
1
𝑛
)
• The influence of upweighting z on the parameters 𝜃

6. INFLUENCE OF A TRAINING POINT
• Influence vs. Euclidean distance

7. INFLUENCE OF A TRAINING POINT
Terry Taewoong Um (terry.t.um@gmail.com)
• The influence of upweighting z on the parameters 𝜃
• The influence of upweighting z on the loss at a test point

8. PERTURBING A TRAINING POINT
Terry Taewoong Um (terry.t.um@gmail.com)
• Move 𝜖 mass from 𝑧 to 𝑧 𝛿
• If x is continuous and 𝛿 is small
• The effect of 𝑧  𝑧 𝛿 on the loss at a test point

9. SUMMARY
Terry Taewoong Um (terry.t.um@gmail.com)
• The influence of 𝑧  𝑧 𝛿 on the loss at a test point
• The influence of upweighting z on the parameters 𝜃
• The influence of upweighting z on the loss at a test point

10. EXAMPLE
Terry Taewoong Um (terry.t.um@gmail.com)
• The influence of upweighting z
• In logistic regression,
• Test : 7, Train : 7 (green), 1 (red)

11. SEVERAL PROBLEMS
Terry Taewoong Um (terry.t.um@gmail.com)
• Calculation of
 Use Hessian-vector products (HVPs)

precompute 𝑠𝑡𝑒𝑠𝑡 by optimizing
or sampling-based approximation

12. SEVERAL PROBLEMS
Terry Taewoong Um (terry.t.um@gmail.com)
• What if is non-convex, so H < 0
 Assuming that is a local minimum point, define a quadratic loss
Then calculate using the above
 empirically working!
• Influence function vs. retraining

13. SEVERAL PROBLEMS
Terry Taewoong Um (terry.t.um@gmail.com)
• What if is non-differentiable?
e.g.) hinge loss
 Use a differentiable variation of the hinge loss

14. APPLICATIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• Understanding model behavior

15. APPLICATIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• Adversarial examples
c.f.) The effect of 𝑧  𝑧 𝛿 on the loss at a test point

16. APPLICATIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• Debugging domain mismatch

17. APPLICATIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• Fixing mislabeled examples