Principal Component Analysis (PCA) is a technique used to reduce the dimensionality of data by transforming correlated variables into a smaller number of uncorrelated variables called principal components. The document discusses PCA concepts like projections, dimensionality reduction, and applications to housing data. It explains how PCA finds the directions of maximum variance in high-dimensional data and projects it onto a new coordinate system.

1. Principal Component Analysis (PCA)

2. Principal Component Analysis (PCA)

3. Principal Component Analysis (PCA)
Luis Serrano

4. 1. Projections

5. Taking a picture

6. Taking a picture

7. Taking a picture

8. Taking a picture

9. Taking a picture

10. Taking a picture

11. Dimensionality Reduction

12. Dimensionality Reduction

13. Dimensionality Reduction

14. Dimensionality Reduction

15. Dimensionality Reduction

16. Dimensionality Reduction

17. Dimensionality Reduction

18. Dimensionality Reduction

19. Dimensionality Reduction

20. Dimensionality Reduction

21. Dimensionality Reduction

22. 2. Application to housing

23. Housing Data

24. Housing Data
Size

25. Housing Data
Size
Number of rooms

26. Housing Data
Size
Number of rooms
Number of bathrooms

27. Housing Data
Size
Number of rooms
Number of bathrooms
Schools around

28. Housing Data
Size
Number of rooms
Number of bathrooms
Schools around
Crime rate

29. Housing Data
Size
Number of rooms
Number of bathrooms
Schools around
Crime rate

30. Housing Data
Size
Number of rooms
Number of bathrooms
Size feature
Schools around
Crime rate

31. Housing Data
Size
Number of rooms
Number of bathrooms
Size feature
Schools around
Crime rate
Location feature

32. 100
200
300
400
1 2 3 4
Number of Rooms
Size
5

33. 100
200
300
400
1 2 3 4
Number of Rooms
Size
5

34. 100
200
300
400
1 2 3 4
Number of Rooms
Size
5

35. 100
200
300
400
1 2 3 4
Number of Rooms
Size
5

36. Size feature

40. 2 dimensions

41. 2 dimensions 1 dimension

42. 2 dimensions 1 dimension
size
number of rooms

43. 2 dimensions 1 dimension
size
number of rooms
size feature

44. Housing Data

45. Housing Data
5 dimensions

46. Housing Data
5 dimensions 2 dimensions

47. Housing Data
Size
Number of rooms
Number of bathrooms
Schools around
Crime rate
5 dimensions 2 dimensions

48. Housing Data
Size
Number of rooms
Number of bathrooms
Size feature
Schools around
Crime rate
5 dimensions 2 dimensions

49. Housing Data
Size
Number of rooms
Number of bathrooms
Size feature
Schools around
Crime rate
Location feature
5 dimensions 2 dimensions

50. 3. Mean, Variance, Covariance

51. Mean

52. Mean

53. Mean

54. Mean
wall

55. Mean
1 2 3 4 5 6
wall

56. Mean
1 2 3 4 5 6
1+2+6
wall

57. Mean
1 2 3 4 5 6
1+2+6
wall
Mean =

58. Mean
1 2 3 4 5 6
1+2+6
wall
Mean =

59. Mean
1 2 3 4 5 6
1+2+6
3wall
Mean =

60. Mean
1 2 3 4 5 6
1+2+6
3
= 3
wall
Mean =

61. Mean
1 2 3 4 5 6
1+2+6
3
= 3
wall
Mean =

62. Variance

63. Variance

64. Variance

65. Variance

66. Variance

67. Variance
1 1

68. Variance
1 1
5 5

69. Variance
1 1
5 5
Variance =

70. Variance
1 1
5 5
12
+02
+12
Variance =

71. Variance
1 1
5 5
12
+02
+12
Variance =

72. Variance
1 1
5 5
12
+02
+12
3
Variance =

73. Variance
1 1
5 5
12
+02
+12
3
= 2/3Variance =

74. Variance
1 1
5 5
12
+02
+12
3
= 2/3Variance =
Variance =

75. Variance
1 1
5 5
12
+02
+12
3
= 2/3Variance =
52
+02
+52
Variance =

76. Variance
1 1
5 5
12
+02
+12
3
= 2/3Variance =
52
+02
+52
Variance =

77. Variance
1 1
5 5
12
+02
+12
3
= 2/3Variance =
52
+02
+52
3
Variance =

78. Variance
1 1
5 5
12
+02
+12
3
= 2/3Variance =
52
+02
+52
3
= 10/3Variance =

79. Mean
1 2 3 4 5 6

80. Mean
2
1 2 3 4 5 6

81. Mean
2
1
1 2 3 4 5 6

82. Mean
2
1
3
1 2 3 4 5 6

83. Mean
2
1
3
1 2 3 4 5 6
Variance =

84. Mean
2
1
3
1 2 3 4 5 6
22
+12
+32
Variance =

85. Mean
2
1
3
1 2 3 4 5 6
22
+12
+32
Variance =

86. Mean
2
1
3
1 2 3 4 5 6
22
+12
+32
3
Variance =

87. Mean
2
1
3
1 2 3 4 5 6
22
+12
+32
3
= 14/3Variance =

88. Variance?

89. Variance?

90. Variance?

91. Variance?

92. Variance?

93. Variance?
x-variance

94. Variance?
x-variance

95. Variance?
x-variance
y-variance

96. Variance?

97. Variance?

98. Variance?

99. Variance?

100. Variance?
22 22

101. Variance?
x-variance =
22
+02
+22
3
= 8/3
22 22

102. Variance?
x-variance =
22
+02
+22
3
= 8/3
22 22

103. Variance?
x-variance =
22
+02
+22
3
= 8/3
22 22
1
1
1
1

104. Variance?
x-variance =
22
+02
+22
3
= 8/3
y-variance =
12
+02
+12
3
= 2/3
22 22
1
1
1
1

105. Covariance
(0,0)
(2,1)
(2,-1)(-2,-1)
(-2,1)

106. Covariance
(0,0)
(2,1)
(2,-1)(-2,-1)
(-2,1)
Product
of
coordinates

107. Covariance
(0,0)
(2,1)
(2,-1)(-2,-1)
(-2,1)
0
Product
of
coordinates

108. Covariance
(0,0)
(2,1)
(2,-1)(-2,-1)
(-2,1)
+2
0
+2
Product
of
coordinates

109. Covariance
(0,0)
(2,1)
(2,-1)(-2,-1)
(-2,1)
+2
-2
0
-2
+2
Product
of
coordinates

110. Covariance
(0,0)
(2,1)
(2,-1)(-2,-1)
(-2,1)
+2
-2
0
-2
+2
Product
of
coordinates

111. Covariance
(0,0)
(2,1)
(2,-1)(-2,-1)
(-2,1)
+2
-2
0
-2
+2
Product
of
coordinates

112. Covariance

113. Covariance
(-2,1)
(2,-1)
-2
-2
(0,0)
0

114. Covariance
(-2,1)
(2,-1)
-2
-2
(0,0)
0
(2,1)
(-2,-1)
2
2(0,0)
0

115. Covariance
covariance =
(-2) + 0 + (-2)
3
= -4/3
(-2,1)
(2,-1)
-2
-2
(0,0)
0
(2,1)
(-2,-1)
2
2(0,0)
0

116. Covariance
covariance =
(-2) + 0 + (-2)
3
= -4/3 covariance =
2 + 0 + 2
3
= 4/3
(-2,1)
(2,-1)
-2
-2
(0,0)
0
(2,1)
(-2,-1)
2
2(0,0)
0

117. Covariance
(0,0) (2,0)
(2,1)
(2,-1)(0,-1)
(0,1)(-2,1)
(-2,0)
(-2,-1)

118. Covariance
(0,0) (2,0)
(2,1)
(2,-1)(0,-1)
(0,1)(-2,1)
(-2,0)
(-2,-1)
-2 2
0
0
0
00
-22

119. Covariance
covariance =
(0,0) (2,0)
(2,1)
(2,-1)(0,-1)
(0,1)(-2,1)
(-2,0)
(-2,-1)
-2 2
0
0
0
00
-22

120. Covariance
covariance =
+ + + + + + + +
9
(0,0) (2,0)
(2,1)
(2,-1)(0,-1)
(0,1)(-2,1)
(-2,0)
(-2,-1)
-2 2 00 000 -22

121. Covariance
covariance =
+ + + + + + + +
9
= 0
(0,0) (2,0)
(2,1)
(2,-1)(0,-1)
(0,1)(-2,1)
(-2,0)
(-2,-1)
-2 2 00 000 -22

122. Covariance

123. Covariance

124. Covariance

125. Covariance

126. Covariance
negative
covariance

127. Covariance
negative
covariance
covariance zero
(or very small)

128. Covariance
negative
covariance
covariance zero
(or very small)
positive
covariance

129. 4. Eigenvectors and Eigenvalues

133. Covariance matrix

134. Covariance matrix

135. Covariance matrix
Var(X)

136. Covariance matrix
Var(X)
Var(Y)

137. Covariance matrix
Var(X)
Var(Y)
Cov(X,Y)
Cov(X,Y)

138. Covariance matrix
Var(X)
Var(Y)
Cov(X,Y)
Cov(X,Y)
=

139. Covariance matrix
Var(X)
Var(Y)
Cov(X,Y)
Cov(X,Y)
=
9
3
4
4

140. Linear Transformations

141. Linear Transformations
9
3
4
4

142. Linear Transformations
9
3
4
4

143. Linear Transformations
9
3
4
4

144. Linear Transformations
9
3
4
4

145. Linear Transformations
9
3
4
4
(x, y)

146. Linear Transformations
9
3
4
4
(x, y)

147. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)

148. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)

149. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)

150. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)

151. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)

152. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)

153. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0)

154. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0)

155. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)

156. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)

157. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)

158. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)

159. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0) (9,4)

160. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0) (9,4)

161. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(9,4)

162. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(9,4)

163. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(9,4)
(4,3)

164. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(9,4)
(4,3)

165. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(9,4)
(4,3)

166. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(9,4)
(4,3)

167. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(9,4)
(4,3)
(-9,-4)

168. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(9,4)
(4,3)
(-9,-4)

169. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(0,-1)
(9,4)
(4,3)
(-9,-4)

170. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(0,-1)
(9,4)
(4,3)
(-9,-4)

171. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(0,-1)
(9,4)
(4,3)
(-9,-4)
(-4,-3)

172. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(0,-1)
(9,4)
(4,3)
(-9,-4)
(-4,-3)

173. Linear Transformations
9
3
4
4
(x, y) (9x+4y, 4x+3y)
(0,0) (0,0)
(1,0)
(0,1)
(-1,0)
(0,-1)
(9,4)
(4,3)
(-9,-4)
(-4,-3)

174. Linear Transformations
9
3
4
4

175. Linear Transformations
9
3
4
4

176. Linear Transformations
9
3
4
4

177. Linear Transformations
9
3
4
4

178. Linear Transformations
9
3
4
4 11

179. Linear Transformations
9
3
4
4 11

180. Linear Transformations
9
3
4
4 11

181. Linear Transformations
9
3
4
4 11

182. Linear Transformations
9
3
4
4 11
1

183. Linear Transformations
9
3
4
4
2
1
11
1

184. Linear Transformations
9
3
4
4
2
1
-1
2
11
1

185. Linear Transformations
9
3
4
4
2
1
-1
2
Eigenvectors
(direction)
11
1

186. Linear Transformations
9
3
4
4
2
1
-1
2 11
Eigenvectors
(direction)
11
1

187. Linear Transformations
9
3
4
4
2
1
-1
2 11 1
Eigenvectors
(direction)
11
1

188. Linear Transformations
9
3
4
4
2
1
-1
2 11 1
Eigenvectors
(direction)
Eigenvalues
(magnitude)
11
1

189. Linear Transformations

190. Linear Transformations
Eigenvectors

191. Linear Transformations
Eigenvectors
Eigenvalues

192. Linear Transformations
2
1
Eigenvectors
Eigenvalues

193. Linear Transformations
2
1
11
Eigenvectors
Eigenvalues

194. Linear Transformations
2
1
11
Eigenvectors
Eigenvalues

195. Linear Transformations
2
1
-1
2
11
Eigenvectors
Eigenvalues

196. Linear Transformations
2
1
-1
2
11 1
Eigenvectors
Eigenvalues

197. Linear Transformations
2
1
-1
2
11 1
Eigenvectors
Eigenvalues

198. Linear Transformations
2
1
-1
2
11 1
Eigenvectors
Eigenvalues

199. Eigenstuff

200. Eigenstuff

201. Eigenstuff

202. Eigenstuff

203. Eigenstuff

204. Eigenstuff

205. Eigenstuff

206. Eigenstuff
9
3
4
4

207. Eigenstuff
9
3
4
4
v

208. Eigenstuff
9
3
4
4
v =

209. Eigenstuff
9
3
4
4
v =

210. Eigenstuff
9
3
4
4
v = v

211. Eigenstuff
9
3
4
4
v = v
Eigenvector

212. Eigenstuff
9
3
4
4
v = v
Eigenvector
Eigenvalue

214. Eigenvalues
9
3
4
4

215. Eigenvalues
Characteristic Polynomial
9
3
4
4

216. Eigenvalues
Characteristic Polynomial
x-9
x-3
-4
-4
9
3
4
4

217. Eigenvalues
Characteristic Polynomial
x-9
x-3
-4
-4
= (x-9)(x-3) - (-4)(-4)
9
3
4
4

218. Eigenvalues
Characteristic Polynomial
x-9
x-3
-4
-4
= (x-9)(x-3) - (-4)(-4) = x2
- 12x + 11
9
3
4
4

219. Eigenvalues
Characteristic Polynomial
x-9
x-3
-4
-4
= (x-9)(x-3) - (-4)(-4) = x2
- 12x + 11
(x-11)(x-1)=
9
3
4
4

220. Eigenvalues
Characteristic Polynomial
x-9
x-3
-4
-4
= (x-9)(x-3) - (-4)(-4) = x2
- 12x + 11
(x-11)(x-1)=
Eigenvalues
9
3
4
4

221. Eigenvalues
Characteristic Polynomial
x-9
x-3
-4
-4
= (x-9)(x-3) - (-4)(-4) = x2
- 12x + 11
(x-11)(x-1)=
Eigenvalues 11
9
3
4
4

222. Eigenvalues
Characteristic Polynomial
x-9
x-3
-4
-4
= (x-9)(x-3) - (-4)(-4) = x2
- 12x + 11
(x-11)(x-1)=
Eigenvalues 11 and
9
3
4
4

223. Eigenvalues
Characteristic Polynomial
x-9
x-3
-4
-4
= (x-9)(x-3) - (-4)(-4) = x2
- 12x + 11
(x-11)(x-1)=
Eigenvalues 11 1and
9
3
4
4

224. Eigenvectors

225. Eigenvectors
9
3
4
4

226. Eigenvectors
9
3
4
4
u
v

227. Eigenvectors
9
3
4
4
u
v
=

228. Eigenvectors
9
3
4
4
u
v
= 11

229. Eigenvectors
9
3
4
4
u
v
=
u
v
11

230. Eigenvectors
9
3
4
4
u
v
=
u
v
11
9
3
4
4

231. Eigenvectors
9
3
4
4
u
v
=
u
v
11
9
3
4
4
u
v

232. Eigenvectors
9
3
4
4
u
v
=
u
v
11
9
3
4
4
u
v
=

233. Eigenvectors
9
3
4
4
u
v
=
u
v
11
9
3
4
4
u
v
= 1

234. Eigenvectors
9
3
4
4
u
v
=
u
v
11
9
3
4
4
u
v
=
u
v
1

235. Eigenvectors
9
3
4
4
u
v
=
u
v
11
9
3
4
4
u
v
=
u
v
1
2
1
u
v
=

236. Eigenvectors
9
3
4
4
u
v
=
u
v
11
9
3
4
4
u
v
=
u
v
1
2
1
u
v
=
-1
2
u
v
=

237. 3blue1brown

238. 4. PCA

239. Principal Component Analysis (PCA)

240. Principal Component Analysis (PCA)

241. Principal Component Analysis (PCA)
=

242. Principal Component Analysis (PCA)
9
=

243. Principal Component Analysis (PCA)
9
3
=

244. Principal Component Analysis (PCA)
9
3
4
4
=

245. Principal Component Analysis (PCA)
Eigenvectors
9
3
4
4
=

246. Principal Component Analysis (PCA)
Eigenvectors
9
3
4
4
=
(direction)

247. Principal Component Analysis (PCA)
Eigenvectors
9
3
4
4
=
Eigenvalues
(direction)

248. Principal Component Analysis (PCA)
Eigenvectors
9
3
4
4
=
Eigenvalues
(direction)
(magnitude)

249. Principal Component Analysis (PCA)
2
1
Eigenvectors
9
3
4
4
=
Eigenvalues
(direction)
(magnitude)

250. Principal Component Analysis (PCA)
2
1
Eigenvectors
9
3
4
4
=
11 Eigenvalues
(direction)
(magnitude)

251. Principal Component Analysis (PCA)
2
1
Eigenvectors
9
3
4
4
=
11 Eigenvalues
(direction)
(magnitude)

252. Principal Component Analysis (PCA)
2
1
-1
2
Eigenvectors
9
3
4
4
=
11 Eigenvalues
(direction)
(magnitude)

253. Principal Component Analysis (PCA)
2
1
-1
2
Eigenvectors
9
3
4
4
=
11 1 Eigenvalues
(direction)
(magnitude)

254. Principal Component Analysis (PCA)
2
1
-1
2
Eigenvectors
9
3
4
4
=
11 1 Eigenvalues
(direction)
(magnitude)

255. Principal Component Analysis (PCA)
2
1
-1
2
Eigenvectors
9
3
4
4
=
11 1 Eigenvalues
(direction)
(magnitude)

256. Principal Component Analysis (PCA)
2
1
-1
2
Eigenvectors
9
3
4
4
=
11 1 Eigenvalues
(direction)
(magnitude)

257. Principal Component Analysis (PCA)
2
1
Eigenvectors
9
3
4
4
=
11 Eigenvalues
(direction)
(magnitude)

258. Principal Component Analysis (PCA)
2
1
Eigenvectors
9
3
4
4
=
11 Eigenvalues
(direction)
(magnitude)

259. Principal Component Analysis (PCA)
2
1
Eigenvectors
9
3
4
4
=
11 Eigenvalues
(direction)
(magnitude)

260. Principal Component Analysis (PCA)
2
1
Eigenvectors
9
3
4
4
=
11 Eigenvalues
(direction)
(magnitude)

261. Principal Component Analysis (PCA)
2
1
Eigenvectors
9
3
4
4
=
11 Eigenvalues
(direction)
(magnitude)

262. Principal Component Analysis (PCA)

263. Principal Component Analysis (PCA)

264. Thank you!

265. Thank you!
Subscribe, like,
share, comment!
youtube.com/c/LuisSerrano

266. Thank you!
@luis_likes_math
Subscribe, like,
share, comment!
youtube.com/c/LuisSerrano