LeCun's DL l2

1. Modules &
Architectures
Yann LeCun
NYU - Courant Institute & Center for Data Science
Facebook AI Research
Deep Learning, NYU Spring 2021

2. Y. LeCun
Deep Learning: Assembling differentiable modules
Deep Learning Definition:
Assembling networks of differentiable
modules and training them with a gradient-
based optimization method.
Any graph of modules is fine
As long as there exist a partial order on the
modules, and we can backpropagate gradient
with respect to the relevant variables
If the graph has loops…
we need to “unroll” them.
Recurrent networks and bakprop through time

3. Y. LeCun
Simple Neural Net
Object-oriented version
Uses predefined nn.Linear class,
(which includes a bias vector)
Uses torch.relu function
State variables are temporary

4. Activation Functions
In PyTorch & how to use them

5. Y. LeCun
ReLU – nn.ReLU()

6. Y. LeCun
RReLU – nn.RReLU()

7. Y. LeCun
LeakyReLU – nn.LeakyReLU()

8. Y. LeCun
PReLU – nn.PReLU()

9. Y. LeCun
Softplus – nn.Softplus()

10. Y. LeCun
ELU – nn.ELU()

11. Y. LeCun
CELU – nn.CELU()
Barron (2017) Continuously differentiable exponential linear units

12. Y. LeCun
CELU – nn.CELU()

13. Y. LeCun
CELU – nn.CELU()

14. Y. LeCun
SELU – nn.SELU()

15. Y. LeCun
SELU – nn.SELU()
Klambauer, Unterthiner, Mayr, Hochreiter (2017) Self-normalizing neural networks

16. Y. LeCun
GELU – nn.GELU()

17. Y. LeCun
ReLU6 – nn.ReLU6()

18. Y. LeCun
Sigmoid – nn.Sigmoid()

19. Y. LeCun
Tanh – nn.Tanh()

20. Y. LeCun
Softsign – nn.Softsign()

21. Y. LeCun
Hardtanh – nn.Hardtanh()

22. Y. LeCun
Threshold – nn.Threshold()

23. Y. LeCun
Tanhshrink – nn.Tanhshrink()

24. Y. LeCun
Softshrink – nn.Softshrink()

25. Y. LeCun
Hardshrink – nn.Hardshrink()

26. Y. LeCun
LogSigmoid – nn.LogSigmoid()

27. Y. LeCun
Softmin – nn.Softmin()

28. Y. LeCun
Softmax – nn.Softmax()

29. Y. LeCun
LogSoftmax – nn.LogSoftmax()

30. Cost Functions
In PyTorch & how to use them

31. Y. LeCun
MSE – nn.MSELoss()

32. Y. LeCun
L1 – nn.L1Loss()

33. Y. LeCun
nn.SmoothL1Loss()

34. Y. LeCun
NLL – nn.NLLLoss()

35. Y. LeCun
CE – nn.CrossEntropyLoss()

36. Y. LeCun
CE – nn.CrossEntropyLoss()

37. Y. LeCun
nn.AdaptiveLogSoftmaxWithLoss()

38. Y. LeCun
KLDiv – nn.KLDivLoss()

39. Y. LeCun
Poisson – nn.PoissonNLLLoss()

40. Y. LeCun
BCE – nn.BCELoss()

41. Y. LeCun
BCE – nn.BCELoss()

42. Y. LeCun
BCE – nn.BCEWithLogitsLoss()

43. Y. LeCun
BCE – nn.BCEWithLogitsLoss()

44. Y. LeCun
Ranking – nn.MarginRankingLoss()

45. Y. LeCun
nn.TripletMarginLoss()

46. Y. LeCun
nn.TripletMarginLoss()

47. Y. LeCun
nn.SoftMarginLoss()

48. Y. LeCun
nn.MultiLabelMarginLoss()

49. Y. LeCun
nn.MultiLabelSoftMarginLoss()

50. Y. LeCun
nn.MultiMarginLoss()

51. Y. LeCun
nn.HingeEmbeddingLoss()

52. Y. LeCun
nn.CosineEmbeddingLoss()

53. Y. LeCun
CTC – nn.CTCLoss()

54. Architectures
Arranging modules

55. Y. LeCun
Multiplicative Modules
Quadratic layer, product units, Sigma-Pi units
Attention module
softmax
+

56. Y. LeCun
Mixture of Experts
Attention module “Switches” expert networks
softmax
+
Expert2
Expert1
Gater

57. Y. LeCun
Parameter transformations
When the parameter vector is the output of a function
G(x,w)
y
x y
C(y,y)
H(u)
u w

58. Y. LeCun
Simple parameter transform: weight sharing
Function H(u) replicates one component of u into multiple
components of w
H is like a “Y” branch.
Gradients are summed in the backprop
The gradients w.r.t. shared parameters are added.
G(x,w)
y
x y
C(y,y)
H(u)
u w

59. Y. LeCun
Shared Weights for Motif Detection
Detecting motifs anywhere on an input
w G(x,w) G(x,w) G(x,w) G(x,w) G(x,w)
MAX