This document outlines the basic building blocks of neural networks. It discusses dense connections, embeddings for text processing, tied weights to reduce parameters, recurrence for sequences, and convolutions for images. These building blocks can be combined to solve complex problems, like predicting house prices from house features, predicting the next word in a sequence of text, image classification to detect outdoor scenes, and image captioning that combines convolutional and recurrent neural networks.

1. THE BUILDING BLOCKS
OF NEURAL NETWORKS
Mat Kelcey

2. BUILDING BLOCK #1

3. PROBLEM: PREDICTING HOUSE PRICES
size (m2) price
190 ?

4. PROBLEM: PREDICTING HOUSE PRICES
size (m2) price
160 $650,000
140 $560,000
200 $820,000
170 $660,000
180 $630,000
210 $800,000
190 ?

5. PROBLEM: PREDICTING HOUSE PRICES

6. PROBLEM: PREDICTING HOUSE PRICES

7. PROBLEM: PREDICTING HOUSE PRICES

8. A SIMPLE MODEL
price = m x size

9. A SIMPLE MODEL
price = m x size
something
we know

10. A SIMPLE MODEL
price = m x size
something
we want to
know

11. A SIMPLE MODEL
price = m x size
something we need
to estimate

12. A GRAPHICAL REPRESENTATION
size price
m
price = m x size

13. A GRAPHICAL REPRESENTATION
price
m1
m2
size
distance
to city
price = m1 x size + m2 x distance

14. A GRAPHICAL REPRESENTATION
house size
distance
block size
age
...
price

15. A GRAPHICAL REPRESENTATION
house size
distance
block size
age
...
price

16. A GRAPHICAL REPRESENTATION

17. A GRAPHICAL REPRESENTATION

18. A GRAPHICAL REPRESENTATION

19. BUILDING BLOCKS
1. dense connections

20. WHAT ABOUT DIFFERENT TYPES OF DATA
block size 62 m2
distance to city 11.0 km
block size 250 m2

21. WHAT ABOUT DIFFERENT TYPES OF DATA
block size 62 m2
distance to city 11.0 km
block size 250 m2
"To be, or not to be."

22. WHAT ABOUT DIFFERENT TYPES OF DATA
block size 62 m2
distance to city 11.0 km
block size 250 m2
"To be, or not to be."

23. WHAT ABOUT TEXT?
"To be, or not to be."

24. PROBLEM: PREDICTING THE NEXT WORD
input: output:
To be or ?

25. REPRESENTING TEXT
{be, not, or, to}
"To be, or not to be."

26. REPRESENTING TEXT
{be, not, or, to}
"To be, or not to be."
{b, e, n, o, r, t}

27. REPRESENTING A WORD
be not or to

28. REPRESENTING A WORD
1 0 0 0
be not or to

29. REPRESENTING A WORD
0 0 1 0
be not or to

30. REPRESENTING A WORD
1 0 0 0
be not or to
-0.3 0.1 0.4

31. REPRESENTING A WORD
0 0 1 0
be not or to
0.1 0.5 -0.9
1 0 0 0
be not or to
-0.3 0.1 0.4

32. BUILDING BLOCKS
1. dense connections
2. embeddings for processing text

33. PREDICTING THE NEXT WORD
0 0 0 1
be not or to
be not or to
0 1 0 0
1 0 0 0
be not or to
0 0 1 0
be not or to

34. TIED WEIGHTS
keep all these
weights the
same!!

35. PARTS OF A NETWORK
input words
represented
as 0s & 1s

36. PARTS OF A NETWORK
input words
represented as
3 values

37. PARTS OF A NETWORK
a combination
of the three
words that
represents the
phrase

38. PARTS OF A NETWORK
the expected
the next word

39. BUILDING BLOCKS
1. dense connections
2. embeddings for processing text
3. tied weights for reducing parameters

40. PREDICTING THE NEXT WORD: v0

41. PREDICTING THE NEXT WORD: v0

42. PREDICTING THE NEXT WORD: v1

43. PREDICTING THE NEXT WORD: v1

44. PREDICTING THE NEXT WORD: v1

45. PREDICTING THE NEXT WORD: v1

46. PREDICTING THE NEXT WORD: v1

47. PREDICTING THE NEXT WORD: v1

48. PREDICTING THE NEXT WORD: v1

49. PREDICTING THE NEXT WORD: v1
tied
weights!

50. BUILDING BLOCKS
1. dense connections
2. embeddings for processing text
3. tied weights for reducing parameters
4. recurrence for handling sequences

51. BUT WHAT ABOUT IMAGES?

52. PROBLEM: "IS OUTDOOR" DETECTOR
input: output:
True

53. "IS OUTDOOR" DETECTOR

54. "IS OUTDOOR" DETECTOR
768 x 512 x 3 = 1,179,648 pixels
512
768
3

55. "IS OUTDOOR" DETECTOR

56. "IS OUTDOOR" DETECTOR
number of parameters
= 768 x 512 x 3
= 1,179,648

57. "IS OUTDOOR" DETECTOR
number of parameters
= 384 x 256 x 3
= 294,912

58. "IS OUTDOOR" DETECTOR
number of parameters
= 384 x 256 x 3
= 294,912

59. BUILDING BLOCKS
1. dense connections
2. embeddings for processing text
3. tied weights for reducing parameters
4. recurrence for handling sequences
5. convolutions for processing images

60. BUILDING WITH BLOCKS

61. PROBLEM: CAPTIONING IMAGES
input: output:
A group of young
people playing a
game of frisbee.
Show and Tell: A Neural Image Caption Generator: Vinyals et al

62. PROBLEM: CAPTIONING IMAGES

63. PROBLEM: CAPTIONING IMAGES
conv
net!

64. PROBLEM: CAPTIONING IMAGES
embeddings!
embeddings!

65. PROBLEM: CAPTIONING IMAGES

66. PROBLEM: CAPTIONING IMAGES
recurrence!!

67. PROBLEM: CAPTIONING IMAGES

68. THANK YOU
For questions or suggestions:
Mat Kelcey
matkelcey@ThoughtWorks.com