3. Inner Product Spaces
Orthogonal and Orthogonal basis
Gram Schmidt Process
Orthogonal complements
Orthogonal projections
Least Squares Approximation
Contents
4. Inner Product Spaces
(1)
(2)
(3)
(4)
(5) if and only if
〉〈〉〈 uvvu ,,
〉〈〉〈〉〈 wuvuwvu ,,,
〉,〈〉,〈 vuvu kk
0, 〉〈 vv
0, 〉〈 vv v 0
Inner product : represented by angle brackets
Let u, v, and w be vectors in a vector space V, and let c be
any scalar. An inner product on V is a function that associates
a real number with each pair of vectors u and v and
satisfies the following axioms (abstraction definition from
the properties of dot product in Theorem 5.3 on Slide 5.12)
(commutative property of the inner product)
(distributive property of the inner product
over vector addition)
(associative property of the scalar multiplication and the
inner product)
,u v〈 〉
,u v〈 〉
5. Note:
dot product (Euclidean inner product for )
, general inner product for a vector space
n
R
V
u v
u v
Note:
A vector space V with an inner product is called an inner
product space
Vector space:
Inner product space:
( , , )V
( , , , , >)V
6. Properties of inner products
Let u, v, and w be vectors in an inner product space V, and
let c be any real number
(1)
(2)
(3)
, , 0 0 v v 0〈 〉〈 〉
, , , u v w u w v w〈 〉〈 〉〈 〉
, ,c cu v u v〈 〉 〈 〉
7. u and v are orthogonal if
Distance between u and v:
vuvuvuvu ,||||),(d
Angle between two nonzero vectors u and v:
0,
||||||||
,
cos
vu
vu 〉〈
Orthogonal:
0, 〉〈 vu
( )u v
Norm (length) of u:
〉〈 uuu ,||||
• The definition of norm (or length), distance, angle, orthogonal,
and normalizing for general inner product spaces closely parallel
to those based on the dot product in Euclidean n-space
8. Properties of norm:
1)
(2) if and only if
(3)
Properties of distance: (the same as the properties for the dot
product in R
n
on Slide 5.9)
(1)
(2) if and only if
(3)
0|||| u
0|||| u 0u
|||||||||| uu cc
0),( vud
0),( vud vu
),(),( uvvu dd
9. Let u and v be vectors in an inner product space V
(1) Cauchy-Schwarz inequality:
(2) Triangle inequality:
(3) Pythagorean theorem:
u and v are orthogonal if and only if
|||||||||||| vuvu
222
|||||||||||| vuvu
|||||||||,| vuvu 〉〈
10. Orthogonal vectors:
Two vectors u and v in R
n
are orthogonal (perpendicular) if
0 vu
Note:
The vector 0 is said to be orthogonal to every vector
Orthogonal and Orthogonal basis
11. Ex : Finding orthogonal vectors
Determine all vectors in R
n
that are orthogonal to u = (4, 2)
0
24
),()2,4(
21
21
vv
vvvu
tv
t
v
21 ,
2
,
2
t
,t t R
v
)2,4(u Let ),( 21 vvv
Sol:
12. Orthogonal basis
Definition:
a,b in V, ab if (a|b)=0.
The zero vector is orthogonal to every vector.
An orthogonal set S is a set s.t. all pairs of distinct vectors
are orthogonal.
An orthonormal set S is an orthogonal set of unit vectors.
Every nonzero finite dimension inner product space has an
orthogonal basis
13. Theorems :
If S={v1,v2,…,vn} is an orthogonal set of nonzero vectorin an
inner product space V then S is linearly independent.
Any orthogonal set of n nonzero vector in Rn is basis for Rn.
If S={v1,v2,…,vn} is an orthonormal basis for an inner product
space V, and u is any vector in V then it can be expressed as a
linear combination of v1,v2,…,vn.
u=<u,v1>v1 + <u,v2>v2 +…+ <u,vn>vn
If S is an orthonormal basis for an n-dimensional inner product
space, and if coordinate vectors of u & v with respect to S are
[u]s=(a1,a2,…,an) and [v]s=(b1,b2,…,bn) then
||u||=√(a ) + (a ) + …+(a )
d(u,v)=√(a1-b1)2 + (a2-b2)2 +…+ (an-bn)2
<u,v>=a1b1 + a2b2 +…+ anbn
=[u]s.[v]s.
2 2 2
1 2
n
14. Gram Schmidt Process
Gram schidt process to orthogonalisation of a set of vectors.
Let {𝑢1, 𝑢2, 𝑢3} be the given set of vector which is basis for
vector space V.
We shall constuct an orthogonal set{𝑣1 𝑣2 𝑣3} of vector of V
which becomes basis for V as under.
Consider the vector space with the Euclidean inner product.
Apply the Gram–Schmidt process to transform the basis
vectors , , into an orthogonal basis ; then normalize the
orthogonal basis vectors to obtain an orthonormal basis .
15. Step 1. Let 𝑉1=𝑈1
Step 2. 𝑣2 = 𝑢2 −
< 𝑢2, 𝑣1> 𝑣1
||𝑣1||
2
Step 3. 𝑣3 =𝑢3 −
< 𝑢3, 𝑣1> 𝑣1
|| 𝑣1||
2 −
< 𝑢3, 𝑣2> 𝑣2
|| 𝑣2||
2 and so on
……………………
And orthonormal basis are
{
𝑣1
||𝑣1||
,
𝑣2
||𝑣2||
,
𝑣3
||𝑣3||
}
18. Orthogonal complements
Let W be a subspace of on inner product space V. A vector u
in V is orthogonal to W if is orthogonal to every vector in W.
The set of all vector in V that are orthogonal to W is called
orthogonal complement of W and is denoted by W.
19. Properties of Orthogonal complements
If W is a subspace of inner product space V than
● A vector u is in W if and only if u is orthogonal to every
spans W.
● The only vector common to W and W is 0.
● W is subspace of v.
● W W = W
20. ● Let W=span {u ,u }
u =(2,0,-1)
u =(4,0,-2)
We have
The homo. system is
AX=0;
3
2
1
x
x
x
x
1
2
1 2
1
2
204
102
Ex : Fine the basis for an orthogonal complement of the subspace of
W span by the vector
u =(2,0,-1)
u =4,0,-2)
22. For inner product spaces:
Let u and v be two vectors in an inner product space V.
If , then the orthogonal projection of u onto v is
given by
0v
,
proj
,
v
u v
u v
v v
2
Consider 0, cos
|| |||| || cos || |||| ||
|| |||| ||
proj
a a a a
a
v
v v v u
u v u v u v u v
v v u v v
u v u v u v
u v
v v v vv
u
v
proj , 0a a vu v
Orthogonal projections : For the dot product function in R
n
, we
define the orthogonal projection of u onto v to be projvu = av (a
scalar multiple of v), and the coefficient a can be derived as
follows
23. Ex : Finding an orthogonal projection in R3
Use the Euclidean inner product in R3
to find the
orthogonal projection of u = (6, 2, 4) onto v = (1, 2, 0)
Sol:
10)0)(4()2)(2()1)(6(, vu
5021, 222
vv
, 10
proj (1, 2 , 0) (2 , 4 , 0)
, 5
v
u v u v
u v v
v v v v
24. Least Squares Approximation :
(A system of linear equations)bx A
11 mnnm
(1) When the system is consistent, we can use the Gaussian
elimination with the back substitution to solve for x
(2) When the system is inconsistent, only the “best possible”
solution of the system can be found, i.e., to find a solution of x
for which the difference (or said the error) between Ax and b is
smallest
Note: the system of linear equations Ax = b is consistent if and only if b is
in the column space of A
25. Least Squares Approximation :
Given a system Ax = b of m linear equations in n unknowns,
the least squares problem is to find a vector x in Rn
that
minimizes the distance between Ax and b, i.e.,
with respect to the Euclidean inner product in R
n
. Such
vector is called a least squares solution of Ax = b
bxA
※ The term least squares comes from the fact that minimizing
is equivalent to minimizing = (Ax – b) ·
(Ax – b), which is a sum of squared errors
A x b
2
A x b
26. ( )
ˆDefine ( ), and the problem to find
ˆsuch that is closest to is equivalent to find
the vector in ( ) closest to , that is proj
m n
n
W
A M
R
A CS A
W CS A
A
CS A
x
x
x
x b
b b
ˆThus proj
ˆ ˆ( proj ) ( ) ( ) ( )
ˆ ( ) ( )
ˆ( )
ˆ
W
W
Τ
Τ
Τ Τ
A
A W A CS A
A CS A NS A
A A
A A A
x b
b b b x b x
b x
b x 0
x b (the n×n linear system of normal equations associated with
Ax = b)
(The nullspace of AT is a solution space of
the homogeneous system ATx=0)
(To find the best solution which should satisfy this equation)ˆx
b
ˆAb x
ˆ projWAx = b
W
27. Note:
The problem of finding the least squares solution of
is equal to the problem of finding an exact solution of the
associated normal system
bx A
bx ΤΤ
AAA ˆ
Theorem associated with Least Squares Approximation solution:
For any linear system , the associated normal system
is consistent, and all solutions of the normal system are least
squares solution of Ax = b. In other words, if W is the column
space of A, and is the least squares solution of Ax = b, then
the orthogonal projection of b on W is , i.e.,
bx A
bx ΤΤ
AAA ˆ
xb ˆproj AW
xˆ
ˆAx
28. Ex : Solving the normal equations
Find the least squares solution of the following system
and find the orthogonal projection of b onto the column space
of A
3
1
0
31
21
11
1
0
c
c
A bx
