1. Algorithms
Sandeep Kumar Poonia
Head Of Dept. CS/IT
B.E., M.Tech., UGC-NET
LM-IAENG, LM-IACSIT,LM-CSTA, LM-AIRCC, LM-SCIEI, AM-UACEE

2. Algorithms
Introduction
Proof By Induction
Asymptotic notation

3. Pre Computer Era:
•
In the pre computer era various models of
computation were proposed. The models worked on a
predefined set of operations. The primary focus in
this era was on “What can be computed and what
cannot be computed?” This field of study was
called Computability Theory.
Sandeep Kumar Poonia

4. Post Computer Era:
• With ultimate computability in terms of Turing
Machines, the primary focus in the post
computer era was Complexity Theory. People
were more interested to find out “How well
can it be computed?”
Sandeep Kumar Poonia

5. Sandeep Kumar Poonia

6. Sandeep Kumar Poonia

7. Sandeep Kumar Poonia

8. As a corollary to the above mentioned result we can easily see that a number n
must be halved floor(log2n) + 1 times to reach 0.
Sandeep Kumar Poonia

9. Algorithms
•An algorithm is any well-defined computational
procedure that takes some value, or set of values, as
input and produces some value, or set of values, as
output.
•An algorithm is thus a sequence of computational
steps that transform the input into the output.
Sandeep Kumar Poonia

10. Algorithms
Sandeep Kumar Poonia

11. Algorithms
Sandeep Kumar Poonia

12. Properties of Algorithms
Sandeep Kumar Poonia

13. Steps to develop an algorithm
Sandeep Kumar Poonia

14. Algorithms as a technology
•Suppose computers were infinitely fast and
•computer memory was free.
Would you have any reason to study algorithms?......YES(See an
example)
let us assume a faster computer (computer A) running insertion sort
against a slower computer (computer B) running merge sort. They each
must sort an array of one million numbers. Suppose that computer A
executes one billion instructions per second and computer B executes
only ten million instructions per second, so that computer A is 100
times faster than computer B in raw computing power.
insertion sort, takes time roughly equal to c1n2 to sort n items,
merge sort, takes time roughly equal to c2n lg n
Sandeep Kumar Poonia

15. Algorithms as a technology
To sort one million numbers,
• Computer A takes
• Computer B takes
By using an algorithm whose running time grows more
slowly, even with a poor compiler, computer B runs 20
times faster than computer A
Sandeep Kumar Poonia

16. Review: Induction
• Suppose
 S(k) is true for fixed constant k
o Often k = 0
 S(n)  S(n+1) for all n >= k
• Then S(n) is true for all n >= k
Sandeep Kumar Poonia

17. Proof By Induction
• Claim:S(n) is true for all n >= k
• Basis:
 Show formula is true when n = k
• Inductive hypothesis:
 Assume formula is true for an arbitrary n
• Step:
 Show that formula is then true for n+1
Sandeep Kumar Poonia

18. Induction Example:
Gaussian Closed Form
• Prove 1 + 2 + 3 + … + n = n(n+1) / 2
 Basis:
o If n = 0, then 0 = 0(0+1) / 2
 Inductive hypothesis:
o Assume 1 + 2 + 3 + … + n = n(n+1) / 2
 Step (show true for n+1):
1 + 2 + … + n + n+1 = (1 + 2 + …+ n) + (n+1)
= n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2
= (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2
Sandeep Kumar Poonia

19. Induction Example:
Geometric Closed Form
• Prove a0 + a1 + … + an = (an+1 - 1)/(a - 1) for
all a  1
 Basis: show that a0 = (a0+1 - 1)/(a - 1)
a0 = 1 = (a1 - 1)/(a - 1)
 Inductive hypothesis:
o Assume a0 + a1 + … + an = (an+1 - 1)/(a - 1)
 Step (show true for n+1):
a0 + a1 + … + an+1 = a0 + a1 + … + an + an+1
= (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1)
Sandeep Kumar Poonia

20. Induction
•
•
We’ve been using weak induction
Strong induction also holds
 Basis: show S(0)
 Hypothesis: assume S(k) holds for arbitrary k <= n
 Step: Show S(n+1) follows
•
Another variation:
 Basis: show S(0), S(1)
 Hypothesis: assume S(n) and S(n+1) are true
 Step: show S(n+2) follows
Sandeep Kumar Poonia

21. Asymptotic Performance
• Here, we care most about asymptotic
performance
 How does the algorithm behave as the problem
size gets very large?
o Running time
o Memory/storage requirements
o Bandwidth/power requirements/logic gates/etc.
Sandeep Kumar Poonia

22. Asymptotic Notation
• By now you should have an intuitive feel for
asymptotic (big-O) notation:
 What does O(n) running time mean? O(n2)?
O(n lg n)?
 How does asymptotic running time relate to
asymptotic memory usage?
• Our first task is to define this notation more
formally and completely
Sandeep Kumar Poonia

23. Analysis of Algorithms
• Analysis is performed with respect to a
•
computational model
We will usually use a generic uniprocessor
random-access machine (RAM)
 All memory equally expensive to access
 No concurrent operations
 All reasonable instructions take unit time
o Except, of course, function calls
 Constant word size
o Unless we are explicitly manipulating bits
Sandeep Kumar Poonia

24. Input Size
• Time and space complexity
 This is generally a function of the input size
o E.g., sorting, multiplication
 How we characterize input size depends:
o Sorting: number of input items
o Multiplication: total number of bits
o Graph algorithms: number of nodes & edges Etc
Sandeep Kumar Poonia

25. Running Time
• Number of primitive steps that are executed
 Except for time of executing a function call most
statements roughly require the same amount of
time
oy=m*x+b
o c = 5 / 9 * (t - 32 )
o z = f(x) + g(y)
• We can be more exact if need be
Sandeep Kumar Poonia

26. Analysis
• Worst case
 Provides an upper bound on running time
 An absolute guarantee
• Average case
 Provides the expected running time
 Very useful, but treat with care: what is “average”?
o Random (equally likely) inputs
o Real-life inputs
Sandeep Kumar Poonia

27. An Example: Insertion Sort
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

28. An Example: Insertion Sort
30
10
40
20
1
2
3
4
i =  j =  key = 
A[j] = 
A[j+1] = 
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

29. An Example: Insertion Sort
30
10
40
20
1
2
3
4
i=2 j=1
A[j] = 30
key = 10
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

30. An Example: Insertion Sort
30
30
40
20
1
2
3
4
i=2 j=1
A[j] = 30
key = 10
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

31. An Example: Insertion Sort
30
30
40
20
1
2
3
4
i=2 j=1
A[j] = 30
key = 10
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

32. An Example: Insertion Sort
30
30
40
20
1
2
3
4
i=2 j=0
A[j] = 
key = 10
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

33. An Example: Insertion Sort
30
30
40
20
1
2
3
4
i=2 j=0
A[j] = 
key = 10
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

34. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=2 j=0
A[j] = 
key = 10
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

35. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=3 j=0
A[j] = 
key = 10
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

36. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=3 j=0
A[j] = 
key = 40
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

37. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=3 j=0
A[j] = 
key = 40
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

38. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=3 j=2
A[j] = 30
key = 40
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

39. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=3 j=2
A[j] = 30
key = 40
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

40. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=3 j=2
A[j] = 30
key = 40
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

41. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=4 j=2
A[j] = 30
key = 40
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

42. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

43. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

44. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 20
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

45. An Example: Insertion Sort
10
30
40
20
1
2
3
4
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 20
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

46. An Example: Insertion Sort
10
30
40
40
1
2
3
4
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

47. An Example: Insertion Sort
10
30
40
40
1
2
3
4
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

48. An Example: Insertion Sort
10
30
40
40
1
2
3
4
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

49. An Example: Insertion Sort
10
30
40
40
1
2
3
4
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

50. An Example: Insertion Sort
10
30
40
40
1
2
3
4
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

51. An Example: Insertion Sort
10
30
30
40
1
2
3
4
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

52. An Example: Insertion Sort
10
30
30
40
1
2
3
4
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

53. An Example: Insertion Sort
10
30
30
40
1
2
3
4
i=4 j=1
A[j] = 10
key = 20
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

54. An Example: Insertion Sort
10
30
30
40
1
2
3
4
i=4 j=1
A[j] = 10
key = 20
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

55. An Example: Insertion Sort
10
20
30
40
1
2
3
4
i=4 j=1
A[j] = 10
key = 20
A[j+1] = 20
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

56. An Example: Insertion Sort
10
20
30
40
1
2
3
4
i=4 j=1
A[j] = 10
key = 20
A[j+1] = 20
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Done!
Sandeep Kumar Poonia

57. Insertion Sort
What is the precondition
InsertionSort(A, n) {
for this loop?
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Sandeep Kumar Poonia

58. Insertion Sort
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
How many times will
}
this loop execute?
Sandeep Kumar Poonia

59. Insertion Sort
Statement
Effort
InsertionSort(A, n) {
for i = 2 to n {
c1n
key = A[i]
c2(n-1)
j = i - 1;
c3(n-1)
while (j > 0) and (A[j] > key) {
c4T
A[j+1] = A[j]
c5(T-(n-1))
j = j - 1
c6(T-(n-1))
}
0
A[j+1] = key
c7(n-1)
}
0
}
T = t2 + t3 + … + tn where ti is number of while expression evaluations for the ith
for loop iteration
Sandeep Kumar Poonia

60. Analyzing Insertion Sort
•
•
T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)
= c8T + c9n + c10
What can T be?
 Best case -- inner loop body never executed
o ti = 1  T(n) is a linear function
 Worst case -- inner loop body executed for all
previous elements
o ti = i  T(n) is a quadratic function
Sandeep Kumar Poonia

61. Analysis
• Simplifications
 Ignore actual and abstract statement costs
 Order of growth is the interesting measure:
o Highest-order term is what counts


Sandeep Kumar Poonia
Remember, we are doing asymptotic analysis
As the input size grows larger it is the high order term that
dominates

62. Upper Bound Notation
• We say InsertionSort’s run time is O(n2)
 Properly we should say run time is in O(n2)
 Read O as “Big-O” (you’ll also hear it as “order”)
• In general a function
 f(n) is O(g(n)) if there exist positive constants c
and n0 such that f(n)  c  g(n) for all n  n0
• Formally
 O(g(n)) = { f(n):  positive constants c and n0 such
that f(n)  c  g(n)  n  n0
Sandeep Kumar Poonia

63. Lower Bound Notation
• We say InsertionSort’s run time is (n)
• In general a function
 f(n) is (g(n)) if  positive constants c and n0 such
that 0  cg(n)  f(n)  n  n0
• Proof:
 Suppose run time is an + b
o Assume a and b are positive (what if b is negative?)
 an  an + b
Sandeep Kumar Poonia

64. Asymptotic Tight Bound
• A function f(n) is (g(n)) if  positive
constants c1, c2, and n0 such that
c1 g(n)  f(n)  c2 g(n)  n  n0
• Theorem
 f(n) is (g(n)) iff f(n) is both O(g(n)) and (g(n))
Sandeep Kumar Poonia

65. Practical Complexity
250
f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Sandeep Kumar Poonia

66. Practical Complexity
500
f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Sandeep Kumar Poonia

67. Practical Complexity
1000
f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n
0
1
Sandeep Kumar Poonia
3
5
7
9
11
13
15
17
19

68. Practical Complexity
5000
4000
f(n) = n
f(n) = log(n)
3000
f(n) = n log(n)
f(n) = n^2
2000
f(n) = n^3
f(n) = 2^n
1000
0
1
Sandeep Kumar Poonia
3
5
7
9
11
13
15
17
19

69. Practical Complexity
10000000
1000000
100000
10000
1000
100
10
1
1
Sandeep Kumar Poonia
4
16
64
256
1024
4096
16384 65536

70. Other Asymptotic Notations
• A function f(n) is o(g(n)) if  positive
•
•
constants c and n0 such that
f(n) < c g(n)  n  n0
A function f(n) is (g(n)) if  positive
constants c and n0 such that
c g(n) < f(n)  n  n0
Intuitively,
 o() is like <
 O() is like 
Sandeep Kumar Poonia
 () is like >
 () is like 
 () is like =