Linear algebra

1. Εφαρμοσμένα Μαθηματικά
Μ. Γιαμαρέλου & Γ. Μπίσκος
Γραμμική Άλγεβρα
Linear Algebra

4. Linear Systems of Algebraic Equations
Solve a system of n simultaneous linear algebraic
equations for n unknowns, x1, x2, … , xn:
a1,1x1 + a1,2 x2 + + a1,n xn = b1
a2,1x1 + a2,1x2 + + a2,nxn = b2
an,1x1 + an,1x2 + + an,n xn = bm
• Problems in science, engineering, economics, etc.
• Systems of non-linear equations → linear equations
• Numerical methods needed for systems larger than 4×4

5. Linear Systems of Algebraic Equations
Solve a system of n simultaneous linear algebraic
equations for n unknowns, x1, x2, … , xn:
a11x1 + a12x2 + + a1n xn = b1
a21x1 + a21x2 + + a2n xn = b2
an1x1 + an1x2 + + ann xn = bm
The system has infinitely many solutions
The system has a single unique solution
The system has no solution

6. Linear Systems in a Matrix/Vector form
Ax = B
a11 a12 a1 j a1n
a21 a22 a2 j a2n
ai1 ai2 aij ain
an1 an2 anj ann
é
ë
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
x1
x2
xi
xn
é
ë
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
=
b1
b2
bi
bn
é
ë
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú

7. Solving the Example in MATLAB
In MATLAB we can solve the example with the single
command x = Ab. To do that we need to type:
>> A = [1 1 1; 2 1 3; 3 1 6];
>> b = [4; 7; 2];
>> x = Ab
x =
19.0000
-7.0000
-8.0000

8. Elimination methods for solving
Linear Systems
• Gaussian elimination
The basic idea is to use a sequence of operations that
converts the original system into a simpler, but
equivalent system that may be solved easily.
a11x1 + a12x2 + + a1n xn = b1
a21x1 + a21x2 + + a2n xn = b2
an1x1 + an1x2 + + ann xn = bm

9. Straightforward Systems
a1,1x1 = b1
a2,1x2 = b2
an,n xn = bn
A =
a1,1 0 0 0
0 a2,2 0 0
0 0 ai,i 0
0 0 0 an,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú

10. Straightforward Systems
a1,1x1 + a1,2 x2 + + a1,n xn = b1
+ a2,1x2 + + a2,n xn = b2
an,n xn = bn
a1,1x1 = b1
a2,1x1 + a2,2 x2 = b2
an,1x1 + an,1x2 + + an,nxn = bn
A =
a1,1 a1,2 a1, j a1,n
0 a2,2 a2 j a2,n
0 0 ai,i ai,n
0 0 0 an,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
A =
a1,1 0 0 0
a2,1 a2,2 0 0
ai,1 ai,2 ai,i 0
an,1 an,2 an, j an,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú

11. Back/Forward-substitution
xn =
bn
an,n
, xi =
bi - ai, j xj
j=i+1
j=n
å
ai,i
Upper triangular:
x1 =
b1
a1,1
, xi =
bi - ai, j xj
j=1
j=i-1
å
ai,i
Lower triangular:
Diagonal: xi =
bi
ai,i
,

12. Elimination methods for solving
Linear Systems
• Gaussian elimination
The basic idea is to use a sequence of operations that
converts the original system into a simpler, but
equivalent system that may be solved easily.
a1,1x1 + a1,2 x2 + + a1,n xn = b1
a2,1x1 + a2,1x2 + + a2,nxn = b2
an,1x1 + an,1x2 + + an,n xn = bm

13. Elementary Row Calculations
• Adding two rows (i.e., two equations)
• Multiplying a scalar to a row
• Combining the two k  c x i + k
aj1x1 + aj2x2 + + ajn xn = bj
+ ak1x1 + ak2x2 + + akn xn = bk
(ai1 + ak1)x1 + (ai2 + ak2)x2 + + (ain + akn )xn = bi + bk
cai1x1 + cai2x2 + + cain xn = cbi
(cai1 + ak1)x1 + (cai2 + ak2)x2 + + (cain + akn )xn = cbi + bk

14. Elementary Row Calculations
¢
A =
a1,1 a1,2 a1,n
a2,1 a2,2 a2,n
ai,1 ai,2 ai,n
(cai,1 + ak,1) (cai,2 + ak,2 ) (cai,n + ak,i )
an,1 an,2 an,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
ú
ú
ú
ú
c x i + k → k

15. Augmented Matrix
[A, b] =
a11 a12 a1 j a1n
a21 a22 a2 j a2n
ai1 ai2 aij ain
an1 an2 anj ann
b1
b2
bi
bn
é
ë
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú

16. Gaussian Elimination
[A, b] =
a11 a12 a1 j a1n
0 a22
(2,1)
a2 j
(2,1)
a2n
(2,1)
0 0 aij
(i,i-1)
ain
(i,i-1)
0 0 0 ann
(n,n-1)
b1
b2
(2,1)
bi
(i.i-1)
bn
(n,n-1)
é
ë
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú

17. Algorithm for Gaussian Elimination
Given that matrix A is n×n we have:
for j = 1:N-1 (iterate over columns from left to right)
for i = j+1:N (iterate over rows below diagonal)
if ai,j = 0 STOP (to avoid division by zero)
λ = ai,j/ai,i
for k = j:N (iterate in row i from column j to right)
ai,k = ai,k – λaj,k
end
bi = bi – λbj
end
end Number of FLOPS = 2/3N3

18. Gauss-Jordan Elimination
[A, b]=
d1,1 0 0 0
0 d2,2 0 0
0 0 di, j 0
0 0 0 an,n
b1
b2
bi
bn
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
ú

19. Partial Pivoting
[A, b]=
a1,1 a1,2 a1, j a1,n
a2,1 - l2,1a1,1 a2,2 - l2,1a1,2 a2, j - l21a1j a2,n - l2,1a1,n
ai,1 ai,2 ai, j ai,n
an,1 an,2 an, j an,n
b1
b2 - l2,1b1
bi
bn
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
l21 =
a21
a11
l21 = ±¥
However if , then
a1,1 = 0

20. Partial Pivoting
[A, b]=
ai,1 ai,2 ai, j ai,n
a2,1 a2,2 a2, j a2,n
a1,1 a1,2 a1, j a1,n
an,1 an,2 an, j an,n
bi
b2
b1
bn
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú

21. Algorithm for GE including partial pivoting
for j = 1:N-1 (iterate over columns from left to right)
select row j such that |aj,i| = max {|ai,i|, |ai+1,i|, . . . , |aN,1|}
if aj,i = 0, no unique solution exists, STOP
if j ≠ i, interchange rows i and j of augmented matrix
for i = j+1:N (iterate over rows below diagonal)
λ = ai,j/ai,I
for k = j:N (iterate in row i from column j to right)
ai,k = ai,k – λaj,k
end
bi = bi – λbj
end
end

22. Matrix Factorization: LU decomposition
• Advantageous when we have systems that have the
same left-hand-side (i.e., only vector b)
• With Gaussian elimination we would have to solve all
the systems using the same number of calculations.

23. • Let us assume that we can decompose A into a
product of a lower L and upper U triangular matrix:
we can obtain the solution by solving two triangular
problems
Matrix Factorization: LU decomposition
Ax = LUx = b
Lc = b
Ux = c

24. LU Decomposition using Gauss
Elimination
a1,1 a1,2 a1, j a1,n
a2,1 a2,2 a2 j a2,n
ai,1 ai,2 ai, j ai,n
an,1 an,2 an, j an,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
=
1 0 0 0
l2,1 1 0 0
li,1 li,2 1 0
ln,1 ln,2 ln, j 1
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
a1,1 a1,2 a1, j a1,n
0 ¢
a2,2 ¢
a2 j ¢
a2,n
0 0 a(i)
i, j a(i)
i,n
0 0 0 a(n)
n,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
ú
A = LU

25. LU Decomposition with Pivoting
• Pivoting may be required for the successful execution of
the Gaussian elimination procedure in finding L and U
• To store all the information about the pivoting we use a
permutation matrix P so that
PA = LU

26. Matrix Inversion
In =
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
é
ë
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú

27. Calculating the Inverse of a Matrix
a1,1 a1,2 a1, j a1,n
a2,1 a2,2 a2 j a2,n
ai,1 ai,2 ai, j ai,n
an,1 an,2 an, j an,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
x1,1 x1,2 x1, j x1,n
x2,1 x2,2 x2 j x2,n
xi,1 xi,2 xi, j xi,n
xn,1 xn,2 xn, j xn,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
=
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
é
ë
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
a1,1 a1,2 a1, j a1,n
a2,1 a2,2 a2 j a2,n
ai,1 ai,2 ai, j ai,n
an,1 an,2 an, j an,n
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
x1,1
x1,2
xi,1
xn,1
é
ë
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
ú
ú
ú
=
1
0
0
0
é
ë
ê
ê
ê
ê
ê
ê
ê
ù
û
ú
ú
ú
ú
ú
ú
ú
1st system:

28. Solving Systems of Linear Eqs. in Matlab

29. Solving the Example in MATLAB
In MATLAB we can solve the example with the single
command x = Ab. To do that we need to type:
>> A = [1 1 1; 2 1 3; 3 1 6];
>> b = [4; 7; 2];
>> x = Ab
x =
19.0000
-7.0000
-8.0000

30. Solving the Example in MATLAB
In MATLAB we can solve the example with the single
command x = Ab. To do that we need to type:
>> A = [1 1 1; 2 1 3; 3 1 6];
>> b = [4; 7; 2];
>> Ainv = inv(A);
>> x = Ainv*b
x =
19.0000
-7.0000
-8.0000

31. Solving the Example in MATLAB
In MATLAB we can solve a linear system using LU
decomposition with the command lu as follows:
>> A = [1 1 1; 2 1 3; 3 1 6];
>> b = [4; 7; 2];
>> [L, U, P] = lu(A)
L = …
U = …
P = …
>> c = L(P*b)
c = …
>> x = Uc
x = …

32. Solving the Example in MATLAB
In MATLAB we can perform LU factorization with the
command lu:
>> A = [1 1 1; 2 1 3; 3 1 6];
>> [L, U, P] = lu(A)
L = …, U = …, P = …
>> b = [4; 7; 2];
>> x = U(L(P*b))
x =
19.0000
-7.0000
-8.0000

33. The Determinant
A =
a11 a12
a21 a22
= a11a22 - a21a12
A =
a11 a12 a13
a21 a22 a23
a31 a32 a33
= a11
a22 a23
a32 a33
- a12
a21 a23
a31 a33
+ a13
a21 a22
a31 a32
If det(A) ≠ 0, (i.e., A is nonsingular) then the system has
a unique solution: Cramer’s method
x1 =
det(Ax1)
det(A)
, x2 =
det(Ax2)
det(A)
, , xn =
det(Axn )
det(A)
,