This document provides an overview of linear algebra concepts including vectors, matrices, and matrix decompositions. It begins with definitions of vectors as ordered tuples of numbers that represent quantities with magnitude and direction. Vectors are elements of vector spaces, which are sets that satisfy properties like closure under addition and scalar multiplication. The document then discusses linear independence, bases, norms, inner products, orthonormal bases, and linear operators. It concludes by stating that these concepts will be applied to image compression.
4. Vectors
What is a vector ?
CS221 Linear Algebra Review 10/11/2011 4 / 37
5. Vectors
What is a vector ?
An ordered tuple of numbers :
u
1
u 2
u=
. . .
un
CS221 Linear Algebra Review 10/11/2011 4 / 37
6. Vectors
What is a vector ?
An ordered tuple of numbers :
u
1
u 2
u=
. . .
un
A quantity that has a magnitude and a direction.
CS221 Linear Algebra Review 10/11/2011 4 / 37
7. Vectors
Vector spaces
More formally a vector is an element of a vector space.
CS221 Linear Algebra Review 10/11/2011 5 / 37
8. Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of its
elements :
CS221 Linear Algebra Review 10/11/2011 5 / 37
9. Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of its
elements :
If vectors u and v ∈ V , then u + v ∈ V .
CS221 Linear Algebra Review 10/11/2011 5 / 37
10. Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of its
elements :
If vectors u and v ∈ V , then u + v ∈ V .
If vector u ∈ V , then α u ∈ V for any scalar α.
CS221 Linear Algebra Review 10/11/2011 5 / 37
11. Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of its
elements :
If vectors u and v ∈ V , then u + v ∈ V .
If vector u ∈ V , then α u ∈ V for any scalar α.
There exists 0 ∈ V such that u + 0 = u for any u ∈ V .
CS221 Linear Algebra Review 10/11/2011 5 / 37
12. Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of its
elements :
If vectors u and v ∈ V , then u + v ∈ V .
If vector u ∈ V , then α u ∈ V for any scalar α.
There exists 0 ∈ V such that u + 0 = u for any u ∈ V .
A subspace is a subset of a vector space that is also a vector space.
CS221 Linear Algebra Review 10/11/2011 5 / 37
13. Vectors
Examples
Euclidean space Rn .
CS221 Linear Algebra Review 10/11/2011 6 / 37
14. Vectors
Examples
Euclidean space Rn .
The span of any set of vectors {u1 , u2 , . . . , un }, dened as :
span (u1 , u2 , . . . , un ) = {α1 u1 + α2 u2 + · · · + αn un | αi ∈ R}
CS221 Linear Algebra Review 10/11/2011 6 / 37
15. Vectors
Subspaces
A line through the origin in Rn (span of a vector).
CS221 Linear Algebra Review 10/11/2011 7 / 37
16. Vectors
Subspaces
A line through the origin in Rn (span of a vector).
A plane in Rn (span of two vectors).
CS221 Linear Algebra Review 10/11/2011 7 / 37
17. Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un }
if it does not lie in their span.
CS221 Linear Algebra Review 10/11/2011 8 / 37
18. Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un }
if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearly
independent of the rest.
CS221 Linear Algebra Review 10/11/2011 8 / 37
19. Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un }
if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearly
independent of the rest.
A basis of a vector space V is a linearly independent set of vectors
whose span is equal to V .
CS221 Linear Algebra Review 10/11/2011 8 / 37
20. Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un }
if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearly
independent of the rest.
A basis of a vector space V is a linearly independent set of vectors
whose span is equal to V .
If a vector space has a basis with d vectors, its dimension is d.
CS221 Linear Algebra Review 10/11/2011 8 / 37
21. Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un }
if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearly
independent of the rest.
A basis of a vector space V is a linearly independent set of vectors
whose span is equal to V .
If a vector space has a basis with d vectors, its dimension is d.
Any other basis of the same space will consist of exactly d vectors.
CS221 Linear Algebra Review 10/11/2011 8 / 37
22. Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un }
if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearly
independent of the rest.
A basis of a vector space V is a linearly independent set of vectors
whose span is equal to V .
If a vector space has a basis with d vectors, its dimension is d.
Any other basis of the same space will consist of exactly d vectors.
The dimension of a vector space can be innite (function spaces).
CS221 Linear Algebra Review 10/11/2011 8 / 37
23. Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
CS221 Linear Algebra Review 10/11/2011 9 / 37
24. Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R that
satises :
CS221 Linear Algebra Review 10/11/2011 9 / 37
25. Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R that
satises :
Homogeneity ||α u|| = |α| ||u||
CS221 Linear Algebra Review 10/11/2011 9 / 37
26. Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R that
satises :
Homogeneity ||α u|| = |α| ||u||
Triangle inequality ||u + v|| ≤ ||u|| + ||v||
CS221 Linear Algebra Review 10/11/2011 9 / 37
27. Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R that
satises :
Homogeneity ||α u|| = |α| ||u||
Triangle inequality ||u + v|| ≤ ||u|| + ||v||
Point separation ||u|| = 0 if and only if u = 0.
CS221 Linear Algebra Review 10/11/2011 9 / 37
28. Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R that
satises :
Homogeneity ||α u|| = |α| ||u||
Triangle inequality ||u + v|| ≤ ||u|| + ||v||
Point separation ||u|| = 0 if and only if u = 0.
Examples :
Manhattan or 1 norm : ||u||1 = i |ui |.
Euclidean or 2 norm : ||u||2 = i ui .
2
CS221 Linear Algebra Review 10/11/2011 9 / 37
29. Vectors
Inner Product
Inner product between u and v : u, v = n uv.
i =1 i i
CS221 Linear Algebra Review 10/11/2011 10 / 37
30. Vectors
Inner Product
Inner product between u and v : u, v = n=1 ui vi .
i
It is the projection of one vector onto the other.
CS221 Linear Algebra Review 10/11/2011 10 / 37
31. Vectors
Inner Product
Inner product between u and v : u, v = n=1 ui vi .
i
It is the projection of one vector onto the other.
Related to the Euclidean norm : u, u = ||u||2 .
2
CS221 Linear Algebra Review 10/11/2011 10 / 37
32. Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaled
by the norms of the vectors :
CS221 Linear Algebra Review 10/11/2011 11 / 37
33. Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaled
by the norms of the vectors :
If u, v 0, u and v are aligned.
CS221 Linear Algebra Review 10/11/2011 11 / 37
34. Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaled
by the norms of the vectors :
If u, v 0, u and v are aligned.
If u, v 0, u and v are opposed.
CS221 Linear Algebra Review 10/11/2011 11 / 37
35. Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaled
by the norms of the vectors :
If u, v 0, u and v are aligned.
If u, v 0, u and v are opposed.
If u, v = 0, u and v are orthogonal.
CS221 Linear Algebra Review 10/11/2011 11 / 37
36. Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and
are orthogonal to each other.
CS221 Linear Algebra Review 10/11/2011 12 / 37
37. Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and
are orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take inner
products.
CS221 Linear Algebra Review 10/11/2011 12 / 37
38. Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and
are orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take inner
products.
Example : x = α1 b1 + α2 b2 .
CS221 Linear Algebra Review 10/11/2011 12 / 37
39. Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and
are orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take inner
products.
Example : x = α1 b1 + α2 b2 .
We compute x, b1 :
x, b1 = α1 b1 + α2 b2 , b1
= α1 b1 , b1 + α2 b2 , b1 By linearity.
= α1 + 0 By orthonormality.
CS221 Linear Algebra Review 10/11/2011 12 / 37
40. Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm and
are orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take inner
products.
Example : x = α1 b1 + α2 b2 .
We compute x, b1 :
x, b1 = α1 b1 + α2 b2 , b1
= α1 b1 , b1 + α2 b2 , b1 By linearity.
= α1 + 0 By orthonormality.
Likewise, α2 = x, b2 .
CS221 Linear Algebra Review 10/11/2011 12 / 37
42. Matrices
Linear Operator
A linear operator L : U → V is a map from a vector space U to
another vector space V that satises :
CS221 Linear Algebra Review 10/11/2011 14 / 37
43. Matrices
Linear Operator
A linear operator L : U → V is a map from a vector space U to
another vector space V that satises :
L (u1 + u2 ) = L (u1 ) + L (u2 ) .
CS221 Linear Algebra Review 10/11/2011 14 / 37
44. Matrices
Linear Operator
A linear operator L : U → V is a map from a vector space U to
another vector space V that satises :
.
L (u1 + u2 ) = L (u1 ) + L (u2 )
L (α u) = α L (u) for any scalar α.
CS221 Linear Algebra Review 10/11/2011 14 / 37
45. Matrices
Linear Operator
A linear operator L : U → V is a map from a vector space U to
another vector space V that satises :
.
L (u1 + u2 ) = L (u1 ) + L (u2 )
L (α u) = α L (u) for any scalar α.
If the dimension n of U and m of V are nite, L can be represented by
an m × n matrix A.
A A An
11 12 ... 1
A 21 A 22 ... A n
2
A=
...
Am 1 Am 2 ... Amn
CS221 Linear Algebra Review 10/11/2011 14 / 37
46. Matrices
Matrix Vector Multiplication
A A An u v
11 12 ... 1 1 1
A 21 A 22 ... A nu
2 2
v 2
Au = =
... . . . . . .
Am 1 Am 2 ... Amn un vm
CS221 Linear Algebra Review 10/11/2011 15 / 37
47. Matrices
Matrix Vector Multiplication
A A An u v
11 12 ... 1 1 1
A 21 A 22 ... A nu
2 2
v 2
Au = =
... . . . . . .
Am 1 Am 2 ... Amn un vm
CS221 Linear Algebra Review 10/11/2011 15 / 37
48. Matrices
Matrix Vector Multiplication
A A An u v
11 12 ... 1 1 1
A 21 A 22 ... A nu
2 2
v 2
Au = =
... . . . . . .
Am 1 Am 2 ... Amn un vm
CS221 Linear Algebra Review 10/11/2011 15 / 37
49. Matrices
Matrix Vector Multiplication
A A An u v
11 12 ... 1 1 1
A 21 A 22 ... A nu
2 2
v 2
Au = =
... . . . . . .
Am 1 Am 2 ... Amn un vm
CS221 Linear Algebra Review 10/11/2011 15 / 37
50. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
51. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
52. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
53. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
54. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
55. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
56. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
57. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
58. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
59. Matrices
Matrix Product
Applying two linear operators A and B denes another linear operator,
which can be represented by another matrix AB.
A A An B B Bp
11 12 ... 1 11 12 ... 1
A 21 A 22 ... B
A n
2 21 B22 ... B p
2
AB =
... ...
Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp
AB AB . . . AB p
11 12 1
AB AB 21 22 . . . AB p 2
=
...
ABm 1 ABm 2 ... ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
60. Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by ipping the rows and
columns.
A11 A21 . . . An1
A12 A22 . . . An2
AT =
...
A1m A2m . . . Anm
CS221 Linear Algebra Review 10/11/2011 17 / 37
61. Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by ipping the rows and
columns.
A11 A21 . . . An1
A12 A22 . . . An2
AT =
...
A1m A2m . . . Anm
Some simple properties :
CS221 Linear Algebra Review 10/11/2011 17 / 37
62. Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by ipping the rows and
columns.
A11 A21 . . . An1
A12 A22 . . . An2
AT =
...
A1m A2m . . . Anm
Some simple properties :
T
AT =A
CS221 Linear Algebra Review 10/11/2011 17 / 37
63. Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by ipping the rows and
columns.
A11 A21 . . . An1
A12 A22 . . . An2
AT =
...
A1m A2m . . . Anm
Some simple properties :
T
AT = A
T
(A + B) = AT + BT
CS221 Linear Algebra Review 10/11/2011 17 / 37
64. Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by ipping the rows and
columns.
A11 A21 . . . An1
A12 A22 . . . An2
AT =
...
A1m A2m . . . Anm
Some simple properties :
T
AT = A
T
(A + B) = AT + BT
T
(AB) = BT AT
CS221 Linear Algebra Review 10/11/2011 17 / 37
65. Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by ipping the rows and
columns.
A11 A21 . . . An1
A12 A22 . . . An2
AT =
...
A1m A2m . . . Anm
Some simple properties :
T
AT = A
T
(A + B) = AT + BT
T
(AB) = BT AT
The inner product for vectors can be represented as u, v = uT v.
CS221 Linear Algebra Review 10/11/2011 17 / 37
67. Matrices
Identity Operator
1 0 ... 0
0 1 . . . 0
I=
...
0 0 ... 1
For any matrix A, AI = A.
CS221 Linear Algebra Review 10/11/2011 18 / 37
68. Matrices
Identity Operator
1 0 ... 0
0 1 . . . 0
I=
...
0 0 ... 1
For any matrix A, AI = A.
I is the identity operator for the matrix product.
CS221 Linear Algebra Review 10/11/2011 18 / 37
69. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
70. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
CS221 Linear Algebra Review 10/11/2011 19 / 37
71. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.
CS221 Linear Algebra Review 10/11/2011 19 / 37
72. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.
Linear subspace of Rm .
CS221 Linear Algebra Review 10/11/2011 19 / 37
73. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.
Linear subspace of Rm .
Row space :
CS221 Linear Algebra Review 10/11/2011 19 / 37
74. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.
Linear subspace of Rm .
Row space :
Span of the rows of A.
CS221 Linear Algebra Review 10/11/2011 19 / 37
75. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.
Linear subspace of Rm .
Row space :
Span of the rows of A.
Linear subspace of Rn .
CS221 Linear Algebra Review 10/11/2011 19 / 37
76. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.
Linear subspace of Rm .
Row space :
Span of the rows of A.
Linear subspace of Rn .
Important fact :
The column and row spaces of any matrix have the same dimension.
CS221 Linear Algebra Review 10/11/2011 19 / 37
77. Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.
Linear subspace of Rm .
Row space :
Span of the rows of A.
Linear subspace of Rn .
Important fact :
The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
78. Matrices
Range and Null space
Let A be an m × n matrix.
CS221 Linear Algebra Review 10/11/2011 20 / 37
79. Matrices
Range and Null space
Let A be an m × n matrix.
Range :
CS221 Linear Algebra Review 10/11/2011 20 / 37
80. Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn .
CS221 Linear Algebra Review 10/11/2011 20 / 37
81. Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn .
Linear subspace of Rm .
CS221 Linear Algebra Review 10/11/2011 20 / 37
82. Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn .
Linear subspace of Rm .
Null space :
CS221 Linear Algebra Review 10/11/2011 20 / 37
83. Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn .
Linear subspace of Rm .
Null space :
u belongs to the null space if Au = 0.
CS221 Linear Algebra Review 10/11/2011 20 / 37
84. Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn .
Linear subspace of Rm .
Null space :
u belongs to the null space if Au = 0.
Subspace of Rn .
CS221 Linear Algebra Review 10/11/2011 20 / 37
85. Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn .
Linear subspace of Rm .
Null space :
u belongs to the null space if Au = 0.
Subspace of Rn .
Every vector in the null space is orthogonal to the rows of A.
The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
86. Matrices
Range and Column Space
Another interpretation of the matrix vector product (Ai is column i of A) :
u
1
u 2
Au = A1 A2 ... An
. . .
un
= u1 A1 + u2 A2 + · · · + un An
CS221 Linear Algebra Review 10/11/2011 21 / 37
87. Matrices
Range and Column Space
Another interpretation of the matrix vector product (Ai is column i of A) :
u
1
u 2
Au = A1 A2 ... An
. . .
un
= u1 A1 + u2 A2 + · · · + un An
The result is a linear combination of the columns of A.
CS221 Linear Algebra Review 10/11/2011 21 / 37
88. Matrices
Range and Column Space
Another interpretation of the matrix vector product (Ai is column i of A) :
u
1
u 2
Au = A1 A2 ... An
. . .
un
= u1 A1 + u2 A2 + · · · + un An
The result is a linear combination of the columns of A.
For any matrix, the range is equal to the column space.
CS221 Linear Algebra Review 10/11/2011 21 / 37
89. Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
CS221 Linear Algebra Review 10/11/2011 22 / 37
90. Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
CS221 Linear Algebra Review 10/11/2011 22 / 37
91. Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
In this case, the action of the matrix is invertible.
CS221 Linear Algebra Review 10/11/2011 22 / 37
92. Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
In this case, the action of the matrix is invertible.
The inversion is also linear and consequently can be represented by
another matrix A−1 .
CS221 Linear Algebra Review 10/11/2011 22 / 37
93. Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
In this case, the action of the matrix is invertible.
The inversion is also linear and consequently can be represented by
another matrix A−1 .
A−1 is the only matrix such that A−1 A = AA−1 = I.
CS221 Linear Algebra Review 10/11/2011 22 / 37
94. Matrices
Orthogonal Matrices
An orthogonal matrix U satises UT U = I.
CS221 Linear Algebra Review 10/11/2011 23 / 37
95. Matrices
Orthogonal Matrices
An orthogonal matrix U satises UT U = I.
Equivalently, U has orthonormal columns.
CS221 Linear Algebra Review 10/11/2011 23 / 37
96. Matrices
Orthogonal Matrices
An orthogonal matrix U satises UT U = I.
Equivalently, U has orthonormal columns.
Applying an orthogonal matrix to two vectors does not change their
inner product :
Uu, Uv = (Uu) Uv
T
= uT UT Uv
= uT v
= u, v
CS221 Linear Algebra Review 10/11/2011 23 / 37
97. Matrices
Orthogonal Matrices
An orthogonal matrix U satises UT U = I.
Equivalently, U has orthonormal columns.
Applying an orthogonal matrix to two vectors does not change their
inner product :
Uu, Uv = (Uu) Uv
T
= uT UT Uv
= uT v
= u, v
Example : Matrices that represent rotations are orthogonal.
CS221 Linear Algebra Review 10/11/2011 23 / 37
98. Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
CS221 Linear Algebra Review 10/11/2011 24 / 37
99. Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.
CS221 Linear Algebra Review 10/11/2011 24 / 37
100. Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.
ab
For a 2 × 2 matrix A = c d :
CS221 Linear Algebra Review 10/11/2011 24 / 37
101. Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.
ab
For a 2 × 2 matrix A = c d :
|A| = ad − bc
CS221 Linear Algebra Review 10/11/2011 24 / 37
102. Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.
ab
For a 2 × 2 matrix A = c d :
|A| = ad − bc
The absolute value of |A| is the area of the parallelogram given by the
rows of A.
CS221 Linear Algebra Review 10/11/2011 24 / 37
103. Matrices
Properties of the Determinant
The denition can be generalized to larger matrices.
CS221 Linear Algebra Review 10/11/2011 25 / 37
104. Matrices
Properties of the Determinant
The denition can be generalized to larger matrices.
|A| = AT .
CS221 Linear Algebra Review 10/11/2011 25 / 37
105. Matrices
Properties of the Determinant
The denition can be generalized to larger matrices.
|A| = AT .
|AB| = |A| |B|
CS221 Linear Algebra Review 10/11/2011 25 / 37
106. Matrices
Properties of the Determinant
The denition can be generalized to larger matrices.
|A| = AT .
|AB| = |A| |B|
|A| = 0 if and only if A is not invertible.
CS221 Linear Algebra Review 10/11/2011 25 / 37
107. Matrices
Properties of the Determinant
The denition can be generalized to larger matrices.
|A| = AT .
|AB| = |A| |B|
|A| = 0 if and only if A is not invertible.
If A is invertible, then A−1 = |A | .
1
CS221 Linear Algebra Review 10/11/2011 25 / 37
109. Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satises :
Au = λu
for some vector u, which we call an eigenvector.
CS221 Linear Algebra Review 10/11/2011 27 / 37
110. Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satises :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ 1) or contracts (λ 1)
eigenvectors but does not rotate them.
CS221 Linear Algebra Review 10/11/2011 27 / 37
111. Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satises :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ 1) or contracts (λ 1)
eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A − λI, which is
consequently not invertible.
CS221 Linear Algebra Review 10/11/2011 27 / 37
112. Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satises :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ 1) or contracts (λ 1)
eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A − λI, which is
consequently not invertible.
The eigenvalues of A are the roots of the equation |A − λI| = 0.
CS221 Linear Algebra Review 10/11/2011 27 / 37
113. Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satises :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ 1) or contracts (λ 1)
eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A − λI, which is
consequently not invertible.
The eigenvalues of A are the roots of the equation |A − λI| = 0.
Not the way eigenvalues are calculated in practice.
CS221 Linear Algebra Review 10/11/2011 27 / 37
114. Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satises :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ 1) or contracts (λ 1)
eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A − λI, which is
consequently not invertible.
The eigenvalues of A are the roots of the equation |A − λI| = 0.
Not the way eigenvalues are calculated in practice.
Eigenvalues and eigenvectors can be complex valued, even if all the
entries of A are real.
CS221 Linear Algebra Review 10/11/2011 27 / 37
115. Matrix Decompositions
Eigendecomposition of a Matrix
Let A be an n × n square matrix with n linearly independent
eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn .
CS221 Linear Algebra Review 10/11/2011 28 / 37
116. Matrix Decompositions
Eigendecomposition of a Matrix
Let A be an n × n square matrix with n linearly independent
eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn .
We dene the matrices :
P= p1 . . . pn
λ1 0 . . . 0
0 λ2 . . . 0
Λ=
...
0 0 ... λn
CS221 Linear Algebra Review 10/11/2011 28 / 37
117. Matrix Decompositions
Eigendecomposition of a Matrix
Let A be an n × n square matrix with n linearly independent
eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn .
We dene the matrices :
P= p1 . . . pn
λ1 0 . . . 0
0 λ2 . . . 0
Λ=
...
0 0 ... λn
A satises AP = PΛ.
CS221 Linear Algebra Review 10/11/2011 28 / 37
118. Matrix Decompositions
Eigendecomposition of a Matrix
Let A be an n × n square matrix with n linearly independent
eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn .
We dene the matrices :
P= p1 . . . pn
λ1 0 . . . 0
0 λ2 . . . 0
Λ=
...
0 0 ... λn
A satises AP = PΛ.
P is full rank. Inverting it yields the eigendecomposition of A :
A = PΛ P− 1
CS221 Linear Algebra Review 10/11/2011 28 / 37
119. Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).
Example : ( 1 1 ).
0 1
CS221 Linear Algebra Review 10/11/2011 29 / 37
120. Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).
Example : ( 1 1 ).
0 1
Trace(A) = Trace(Λ) = n=1 λi .
i
CS221 Linear Algebra Review 10/11/2011 29 / 37
121. Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).
Example : ( 1 1 ).
0 1
Trace(A) = Trace(Λ) = n=1 λi .
i
|A| = |Λ| = n=1 λi .
i
CS221 Linear Algebra Review 10/11/2011 29 / 37
122. Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).
Example : ( 1 1 ).
0 1
Trace(A) = Trace(Λ) = n=1 λi .
i
|A| = |Λ| = n=1 λi .
i
The rank of A is equal to the number of nonzero eigenvalues.
CS221 Linear Algebra Review 10/11/2011 29 / 37
123. Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).
Example : ( 1 1 ).
0 1
Trace(A) = Trace(Λ) = n=1 λi .
i
|A| = |Λ| = n=1 λi .
i
The rank of A is equal to the number of nonzero eigenvalues.
If λ is a nonzero eigenvalue of A, λ is an eigenvalue of A−1 with the
1
same eigenvector.
CS221 Linear Algebra Review 10/11/2011 29 / 37
124. Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).
Example : ( 1 1 ).
0 1
Trace(A) = Trace(Λ) = n=1 λi .
i
|A| = |Λ| = n=1 λi .
i
The rank of A is equal to the number of nonzero eigenvalues.
If λ is a nonzero eigenvalue of A, λ is an eigenvalue of A−1 with the
1
same eigenvector.
The eigendecomposition makes it possible to compute matrix powers
eciently :
m
Am = PΛP−1 = PΛP−1 PΛP−1 . . . PΛP−1 = PΛm P−1 .
CS221 Linear Algebra Review 10/11/2011 29 / 37
127. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
CS221 Linear Algebra Review 10/11/2011 31 / 37
128. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
The eigenvectors of symmetric matrices are orthonormal.
CS221 Linear Algebra Review 10/11/2011 31 / 37
129. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
The eigenvectors of symmetric matrices are orthonormal.
Consequently, the eigendecomposition becomes :
A = UΛUT for Λ real and U orthogonal.
CS221 Linear Algebra Review 10/11/2011 31 / 37
130. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
The eigenvectors of symmetric matrices are orthonormal.
Consequently, the eigendecomposition becomes :
A = UΛUT for Λ real and U orthogonal.
The eigenvectors of A are an orthonormal basis for the column space
(and the row space, since they are equal).
CS221 Linear Algebra Review 10/11/2011 31 / 37
131. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUT u can be
decomposed into :
CS221 Linear Algebra Review 10/11/2011 32 / 37
132. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUT u can be
decomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).
CS221 Linear Algebra Review 10/11/2011 32 / 37
133. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUT u can be
decomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).
2 Scaling of each coecient Ui , u by the corresponding eigenvalue
(multiplication by Λ).
CS221 Linear Algebra Review 10/11/2011 32 / 37
134. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUT u can be
decomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).
2 Scaling of each coecient Ui , u by the corresponding eigenvalue
(multiplication by Λ).
3 Linear combination of the eigenvectors scaled by the resulting
coecient (multiplication by U).
CS221 Linear Algebra Review 10/11/2011 32 / 37
135. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUT u can be
decomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).
2 Scaling of each coecient Ui , u by the corresponding eigenvalue
(multiplication by Λ).
3 Linear combination of the eigenvectors scaled by the resulting
coecient (multiplication by U).
Summarizing :
n
Au = λi Ui , u Ui
i =1
CS221 Linear Algebra Review 10/11/2011 32 / 37
136. Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUT u can be
decomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).
2 Scaling of each coecient Ui , u by the corresponding eigenvalue
(multiplication by Λ).
3 Linear combination of the eigenvectors scaled by the resulting
coecient (multiplication by U).
Summarizing :
n
Au = λi Ui , u Ui
i =1
It would be great to generalize this to all matrices...
CS221 Linear Algebra Review 10/11/2011 32 / 37
137. Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
CS221 Linear Algebra Review 10/11/2011 33 / 37
138. Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space of
A.
CS221 Linear Algebra Review 10/11/2011 33 / 37
139. Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space of
A.
The columns of V are an orthonormal basis for the row space of A.
CS221 Linear Algebra Review 10/11/2011 33 / 37
140. Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space of
A.
The columns of V are an orthonormal basis for the row space of A.
Σ is diagonal. The entries on its diagonal σi = Σii are positive real
numbers, called the singular values of A.
CS221 Linear Algebra Review 10/11/2011 33 / 37
141. Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space of
A.
The columns of V are an orthonormal basis for the row space of A.
Σ is diagonal. The entries on its diagonal σi = Σii are positive real
numbers, called the singular values of A.
The action of A on a vector u can be decomposed into :
n
Au = σi Vi , u Ui
i =1
CS221 Linear Algebra Review 10/11/2011 33 / 37
142. Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix AT A are equal to the
square of the singular values of A :
AT A = VΣUT UΣVT = VΣ2 VT
CS221 Linear Algebra Review 10/11/2011 34 / 37
143. Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix AT A are equal to the
square of the singular values of A :
AT A = VΣUT UΣVT = VΣ2 VT
The rank of a matrix is equal to the number of nonzero singular
values.
CS221 Linear Algebra Review 10/11/2011 34 / 37
144. Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix AT A are equal to the
square of the singular values of A :
AT A = VΣUT UΣVT = VΣ2 VT
The rank of a matrix is equal to the number of nonzero singular
values.
The largest singular value σ1 is the solution to the optimization
problem :
||Ax||2
σ1 = max
x=0 ||x||2
CS221 Linear Algebra Review 10/11/2011 34 / 37
145. Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix AT A are equal to the
square of the singular values of A :
AT A = VΣUT UΣVT = VΣ2 VT
The rank of a matrix is equal to the number of nonzero singular
values.
The largest singular value σ1 is the solution to the optimization
problem :
||Ax||2
σ1 = max
x=0 ||x||2
It can be veried that the largest singular value satises the properties
of a norm, it is called the spectral norm of the matrix.
CS221 Linear Algebra Review 10/11/2011 34 / 37
146. Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix AT A are equal to the
square of the singular values of A :
AT A = VΣUT UΣVT = VΣ2 VT
The rank of a matrix is equal to the number of nonzero singular
values.
The largest singular value σ1 is the solution to the optimization
problem :
||Ax||2
σ1 = max
x=0 ||x||2
It can be veried that the largest singular value satises the properties
of a norm, it is called the spectral norm of the matrix.
In statistics analyzing data with the singular value decomposition is
called Principal Component Analysis.
CS221 Linear Algebra Review 10/11/2011 34 / 37
148. Application : Image Compression
Compression using the Singular Value Decomposition
The singular value decomposition can be viewed as a sum of rank 1
matrices :
CS221 Linear Algebra Review 10/11/2011 36 / 37
149. Application : Image Compression
Compression using the Singular Value Decomposition
The singular value decomposition can be viewed as a sum of rank 1
matrices :
If some of the singular values are very small, we can discard them.
CS221 Linear Algebra Review 10/11/2011 36 / 37
150. Application : Image Compression
Compression using the Singular Value Decomposition
The singular value decomposition can be viewed as a sum of rank 1
matrices :
If some of the singular values are very small, we can discard them.
This is a form of lossy compression.
CS221 Linear Algebra Review 10/11/2011 36 / 37
151. Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent the
matrix with less parameters.
CS221 Linear Algebra Review 10/11/2011 37 / 37
152. Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent the
matrix with less parameters.
For a 512 × 512 matrix :
CS221 Linear Algebra Review 10/11/2011 37 / 37
153. Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent the
matrix with less parameters.
For a 512 × 512 matrix :
Full representation : 512 × 512 =262 144
CS221 Linear Algebra Review 10/11/2011 37 / 37
154. Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent the
matrix with less parameters.
For a 512 × 512 matrix :
Full representation : 512 × 512 =262 144
Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250
CS221 Linear Algebra Review 10/11/2011 37 / 37
155. Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent the
matrix with less parameters.
For a 512 × 512 matrix :
Full representation : 512 × 512 =262 144
Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250
Rank 40 approximation : 512 × 40 + 40 + 512 × 40 =41 000
CS221 Linear Algebra Review 10/11/2011 37 / 37
156. Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent the
matrix with less parameters.
For a 512 × 512 matrix :
Full representation : 512 × 512 =262 144
Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250
Rank 40 approximation : 512 × 40 + 40 + 512 × 40 =41 000
Rank 80 approximation : 512 × 80 + 80 + 512 × 80 =82 000
CS221 Linear Algebra Review 10/11/2011 37 / 37
