2. 2-1
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 1-1
What is an algorithm?
An algorithm is a sequence of unambiguous instructions
for solving a problem, i.e., for obtaining a required
output for any legitimate input in a finite amount of
time.
“computer”
problem
algorithm
input output
3. 2-2
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 1-2
Algorithm
• Representable in various forms
• Unambiguity/clearness
• Effectiveness
• Finiteness/termination
• Correctness
4. 2-3
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 1-3
Historical Perspective
Euclid’s algorithm for finding the greatest common divisor
Muhammad ibn Musa al-Khwarizmi – 9th century
mathematician
www.lib.virginia.edu/science/parshall/khwariz.html
5. 2-4
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 1-4
Euclid’s Algorithm
Problem: Find gcd(m,n), the greatest common divisor of two
nonnegative, not both zero integers m and n
Examples: gcd(60,24) = 12, gcd(60,0) = 60,
Euclid’s algorithm is based on repeated application of equality
gcd(m,n) = gcd(n, m mod n)
until the second number becomes 0, which makes the problem
trivial.
Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12
gcd(0,0) = ?
6. 2-5
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 1-5
Two descriptions of Euclid’s algorithm
Step 1 If n = 0, return m and stop; otherwise go to Step 2
Step 2 Divide m by n and assign the value of the remainder to r
Step 3 Assign the value of n to m and the value of r to n. Go to
Step 1.
while n ≠ 0 do
r ← m mod n
m← n
n ← r
return m
7. 2-6
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Analysis of algorithms
Issues:
• correctness
• time efficiency (how fast?)
• space efficiency (how much memory required?)
• optimality
Approaches:
• theoretical analysis
• empirical analysis
8. 2-7
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Theoretical analysis of time efficiency
Time efficiency is a function of input size.
It is analyzed by determining the number of repetitions of the
basic operation of the algorithm.
Basic operation: the operation that contributes the most towards
the running time of the algorithm
T(n) ≈ copC(n)
running time execution time
for basic operation
or cost
Number of times
basic operation is
executed
input size Note: Different basic
operations may cost
differently!
9. 2-8
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Input size and basic operation examples
Problem Input size measure Basic operation
Searching for key in a
list of n items
Number of list’s items, i.e.
n
Key comparison
Multiplication of two
matrices
Matrix dimensions or total
number of elements
Multiplication of two
numbers
Checking primality of a
given integer n
n’size = number of digits
(in binary representation)
Division
Typical graph problem #vertices and/or edges
Visiting a vertex or
traversing an edge
10. 2-9
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Best-case, average-case, worst-case
For some algorithms, the efficiency (running time) can be different
for different inputs of the same size:
Worst case: Cworst(n) – maximum over inputs of size n
Best case: Cbest(n) – minimum over inputs of size n
Average case: Cavg(n) – “average” over inputs of size n
• Average refers to the number of operations required for a typical input, on
average.
• NOT the average of worst and best case
11. 2-10
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Example: Sequential search
Worst case
Best case
Average case
n key comparisons
1 comparisons
(n+1)/2, assuming K is in A
12. 2-11
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Types of formulas for basic operation’s count
Exact formula
e.g., C(n) = n(n-1)/2
Formula indicating order of growth with specific
multiplicative constant
e.g., C(n) ≈ 0.5 n2
Formula indicating order of growth with unknown
multiplicative constant
e.g., C(n) ≈ cn2
13. 2-12
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Order of growth
How slow/fast the algorithm will be if the input size is
changed?
Order of growth is defined within a constant multiple as
n→∞
• For small values of n, the efficiency does not really matter.
Example:
• How much faster will algorithm run on computer that is
twice as fast?
• How much longer does it take to solve problem of double
input size?
14. 2-13
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Values of some important functions as n
15. 2-14
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Asymptotic order of growth
A way of comparing functions that ignores constant factors and
small input sizes (because?)
t(n) is O(g(n)):
class of functions t(n) that grow no faster than g(n)
t(n) is Θ(g(n)):
class of functions t(n) that grow at same rate as g(n)
f(n) is Ω(g(n)):
class of functions t(n) that grow at least as fast as g(n)
16. 2-15
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
O-notation
Definition: t(n) is in O(g(n)), denoted t(n) O(g(n)), if order of
growth of t(n) ≤ order of growth of g(n) (within constant
multiple), i.e., there exist positive constant c and non-negative
integer n0 such that t(n) ≤ c g(n) for every n ≥ n0
Examples:
10n is in O(n2)
5n+20 is in O(n)
17. 2-16
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
-notation
Definition: A function t(n) is said to be in (g(n)), denoted t(n)
(g(n)), if t(n) is bounded below by some constant multiple of g(n)
for all large n, i.e., if there exist some positive constant c and some
nonnegative integer n0 such that
t(n) cg(n) for all n n0
Exercises: prove the following
• 10n2 (n2)
• 0.3n2 - 2n (n2)
• 0.1n3 (n2)
18. 2-17
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
-notation
Definition
A function t(n) is said to be in (g(n)), denoted t(n) (g(n)), if t(n)
is bounded both above and below by some positive constant
multiples of g(n) for all large n, i.e., if there exist some positive
constant c1 and c2 and some nonnegative integer n0 such that
c2 g(n) t(n) c1 g(n) for all n n0
Exercises: prove the following
• 10n2 (n2)
• 0.3n2 - 2n (n2)
• (1/2)n(n+1) (n2)
19. 2-18
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
(g(n)), functions that grow at least as fast as g(n)
(g(n)), functions that grow at the same rate as g(n)
O(g(n)), functions that grow no faster than g(n)
g(n)
>=
<=
=
20. 2-20
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Some properties of asymptotic order of growth
f(n) O(f(n))
f(n) O(g(n)) iff g(n) (f(n))
If f (n) O(g (n)) and g(n) O(h(n)) , then f(n) O(h(n))
Note similarity with a ≤ b ≤ c
If f1(n) O(g1(n)) and f2(n) O(g2(n)) , then
f1(n) + f2(n) O(max{g1(n), g2(n)})
Also, 1in (f(i)) = (1in f(i))
Exercise: Can you prove
these properties?
21. 2-21
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Establishing order of growth using limits
lim T(n)/g(n) =
0 order of growth of T(n) < order of growth of g(n)
c > 0 order of growth of T(n) = order of growth of g(n)
∞ order of growth of T(n) > order of growth of g(n)
Examples:
• 10n vs. n2
• n(n+1)/2 vs. n2
n→∞
22. 2-23
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Orders of growth of some important functions
All logarithmic functions loga n belong to the same class
(log n) no matter what the logarithm’s base a > 1 is
because
All polynomials of the same degree k belong to the same class:
aknk + ak-1nk-1 + … + a0 (nk)
Exponential functions an have different orders of growth for
different a’s
order log n < order n (>0) < order an < order n! < order nn
log𝑎 𝑛 = log𝑏 𝑛 / log𝑏 𝑎
23. 2-24
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Basic asymptotic efficiency classes
1 constant
log n logarithmic
n linear
n log n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial
24. 2-25
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Basic asymptotic efficiency classes
25. 2-26
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Time efficiency of nonrecursive algorithms
General Plan for Analysis
Decide on parameter n indicating input size
Identify algorithm’s basic operation
Determine worst, average, and best cases for input of size n
Set up a sum for the number of times the basic operation is
executed
Simplify the sum using standard formulas and rules (see
Appendix A)
26. 2-27
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Useful summation formulas and rules
lin1 = 1+1+…+1 = n - l + 1
In particular, lin1 = n - 1 + 1 = n (n)
1in i = 1+2+…+n = n(n+1)/2 n2/2 (n2)
1in i2 = 12+22+…+n2 = n(n+1)(2n+1)/6 n3/3 (n3)
0in ai = 1 + a +…+ an = (an+1 - 1)/(a - 1) for any a 1
In particular, 0in 2i = 20 + 21 +…+ 2n = 2n+1 - 1 (2n )
(ai ± bi ) = ai ± bi ; cai = cai ; liuai = limai + m+1iuai
1𝑘
+ 2𝑘
+ 3𝑘
+ ⋯ + 𝑛𝑘
≤ 𝑛𝑘
+ 𝑛𝑘
+ 𝑛𝑘
+ ⋯ + 𝑛𝑘
= 𝑛𝑘+1
∈ Θ(𝑛𝑘+1
)
27. 2-28
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Example 1: Maximum element
C(n) = 1in-1 1 = n-1 = (n) comparisons
28. 2-29
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Example 2: Element uniqueness problem
𝐶 𝑛 =
𝑖=0
𝑛−2
𝑗=𝑖+1
𝑛−1
1 =
𝑖=0
𝑛−2
𝑛 − 𝑖 − 1 = 𝑛 − 1 + 𝑛 − 2 + ⋯ + 1
= 𝑛(𝑛 − 1)/2 = 𝜃(𝑛2
)
29. 2-30
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Example 3: Matrix multiplication
𝐶 𝑛 =
𝑖=0
𝑛−1
𝑗=0
𝑛−1
𝑘=0
𝑛−1
1 =
𝑖=0
𝑛−1
𝑗=0
𝑛−1
𝑛 =
𝑖=0
𝑛−1
𝑛2
= 𝑛3
multiplications
also 𝑛3 additions
30. 2-31
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Example 4: Counting binary digits
It cannot be investigated the way the previous examples are.
The halving game: Find integer i such that n/ ≤ 1.
Answer: i ≥ log n. So, T(n) = (log n) divisions.
Another solution: Using recurrence relations.
2𝑖
31. 2-32
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Plan for Analysis of Recursive Algorithms
Decide on a parameter indicating an input’s size.
Identify the algorithm’s basic operation.
Check whether the number of times the basic op. is executed may
vary on different inputs of the same size. (If it may, the worst,
average, and best cases must be investigated separately.)
Set up a recurrence relation with an appropriate initial condition
expressing the number of times the basic op. is executed.
Solve the recurrence (or, at the very least, establish its solution’s
order of growth) by backward substitutions or another method.
32. 2-33
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Example 1: Recursive evaluation of n!
Definition: n ! = 1 2 … (n-1) n for n ≥ 1 and 0! = 1
Recursive definition of n!: F(n) = F(n-1) n for n ≥ 1 and
F(0) = 1
Size:
Basic operation:
Recurrence relation:
n
multiplication
M(n) = M(n-1) + 1
M(0) = 0
33. 2-34
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Solving the recurrence for M(n)
M(n) = M(n-1) + 1, M(0) = 0
M(n) = M(n-1) + 1
= (M(n-2) + 1) + 1 = M(n-2) + 2
= (M(n-3) + 1) + 2 = M(n-3) + 3
…
= M(n-i) + i
= M(0) + n
= n
The method is called backward substitution.
M(n) = M(n-1)+1
M(n-1) = M(n-2)+1
M(n-2) = M(n-3)+1
M(n-3) = M(n-4)+1
…
We can get rid of M(n-i) if we set i=n,
because we know the value of M(0).
34. 2-35
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Example 2: The Tower of Hanoi Puzzle
1
2
3
Recurrence for number of moves:
M(n) = M(n-1) + 1+ M(n-1)
35. 2-36
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Solving the recurrence for M(n)
M(n) = 2M(n-1) + 1, M(0) = 1
M(n) = 2M(n –1) + 1
= 2[2M(n–2) + 1]+ 1 = 22M(n–2)+2+1
= 22[2M(n–3) + 1]+2+1 = 23M(n–3)+22+2+1
= 23[2M(n–4)+1]+22+2+1 = 24M(n–4)+23+22+2+1
…
= 2iM(n–i)+2i–1+ 2i–2+ 2i–3+… 23+22+2+1
= 2nM(0)+ 2n–1+ 2n–2+ 2n–3+… 23+22+2+1
= 2n+ 2n–1+ 2n–2+ 2n–3+… 23+22+2+1 = 2n+1 – 1
M(n) = 2M(n-1)+1
M(n-1) = 2M(n-2)+1
M(n-2) = 2M(n-3)+1
M(n-3) = 2M(n-4)+1
…
We can get rid of M(n-i) if
we set i=n, because we
know the value of M(0).
36. 2-37
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Example 3: Counting #bits
A( ) = A( ) + 1, A( ) = 0 (using the Smoothness Rule)
= (A( ) + 1) + 1 = A( ) + 2
= A( ) + i
= A( ) + k = k + 0
=
2𝑘
2𝑘−1 20
2𝑘−2
log2 𝑛
2𝑘−2
2𝑘−𝑖
2𝑘−𝑘
A(n) = A( ) + 1, A(1) = 0
𝑛/2
Assume n = 2k
i.e., k = log2 n
37. 2-38
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Smoothness Rule
Let f(n) be a nonnegative function defined on the set of
natural numbers. f(n) is called smooth if it is eventually
nondecreasing and
f(2n) ∈ Θ (f(n))
• Functions that do not grow too fast, including logn, n, nlogn,
and n where >=0 are smooth.
Smoothness rule
Let T(n) be an eventually nondecreasing function and f(n) be
a smooth function. If
T(n) ∈ Θ (f(n)) for values of n that are powers of b,
where b>=2, then
T(n) ∈ Θ (f(n)) for any n.
38. 2-39
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Empirical analysis of time efficiency
Many algorithms cannot be analyzed mathematically.
The alternative way is the Empirical Analysis.
The steps are:
• Select a specific (typical) sample of inputs
• Decide on the efficiency metric (execution time or operation
count)
• Implement the algorithm
• Run the code and record the observed data (time or number
of operations)
• Analyze the data
39. 2-40
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Empirical analysis of time efficiency
The recorded time is usually not accurate. In general,
we repeat the experiment many times and take
average (or median) value.
Empirical analysis is usually done on several inputs of
different sizes.
By measuring the time at inputs of size N, 2N, 3N, …
and calculating their ratios, we can guess the
efficiency class of the algorithm (linear, log, square
etc).
40. 2-41
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Algorithm Visualization
We have studied mathematical and empirical analysis
of algorithms. A third way is Algorithm Visualization.
It uses static images or animations to show the
performance of algorithm.
Initial and final states
of a sorting algorithm
using bar chart
Initial and final states
of a sorting algorithm
using scatter plot
41. 2-42
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Summary
There are two kinds of algorithm efficiency: time efficiency and space
efficiency. Time efficiency indicates how fast the algorithm runs; space
efficiency deals with the extra space it requires.
An algorithm’s time efficiency is principally measured as a function of its
input size by counting the number of times its basic operation is executed. A
basic operation is the operation that contributes the most to running time.
Typically, it is the most time-consuming operation in the algorithm’s
innermost loop.
For some algorithms, the running time can differ considerably for inputs of
the same size, leading to worst-case efficiency, average-case efficiency, and
best-case efficiency.
The established framework for analyzing time efficiency is primarily
grounded in the order of growth of the algorithm’s running time as its input
size goes to infinity.
The notations O, Omega, and Theta are used to indicate and compare the
asymptotic orders of growth of functions expressing algorithm efficiencies.
The efficiencies of a large number of algorithms fall into the following few
classes: constant, logarithmic, linear, linearithmic, quadratic, cubic, and
exponential.
42. 2-43
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Summary
The main tool for analyzing the time efficiency of a nonrecursive algorithm is
to set up a sum expressing the number of executions of its basic operation
and ascertain the sum’s order of growth.
The main tool for analyzing the time efficiency of a recursive algorithm is to
set up a recurrence relation expressing the number of executions of its basic
operation and ascertain the solution’s order of growth.
Empirical analysis of an algorithm is performed by running a program
implementing the algorithm on a sample of inputs and analyzing the data
observed (the basic operation’s count or physical running time). This often
involves generating pseudorandom numbers. The applicability to any
algorithm is the principal strength of this approach; the dependence of
results on the particular computer and instance sample is its main weakness.
Algorithm visualization is the use of images to convey useful information
about algorithms. The two principal variations of algorithm visualization are
static algorithm visualization and dynamic algorithm visualization (also called
algorithm animation).