ICT role in 21st century education and it's challenges.
Itc (ece) by nitin mittal
1. INFORMATION THEORY
AND
CODING
Nitin Mittal
Head of Department
Electronics and Communication Engineering
Modern Institute of Engineering & Technology
Mohri, Kurukshetra
BHARAT PUBLICATIONS
135-A, Santpura Road, Yamuna Nagar - 135001
3. PREFACE
The present volume is the outcome of my experience in teaching of the information
theory to the undergraduate classes for the last few years and the exposure to the problems
faced by the students in grasping the abstract nature of the subject. This experience is the
foundation and, I hope, the strength of the text. Earnest efforts have been exerted to present
the subject matter in a well–knit manner so as not only to stimulate the interest of the students
but also to provide them with an insight into the complexities of a subject of great intrinsic
beauty.
The book is intended to serve as a text for undergraduate students especially those
opting for a course in Electronics and Communication Engineering. However, post graduate
students will find it equally useful.
This book offers a comprehensive review of the Information Theory and Coding. The
main text can be divided into four sections on Probability Theory, Information Theory, Source
Coding and Error Control Coding. Fairly sufficient ground has been covered in all the four
sections. Information theory is the study of achievable bounds for communication and is largely
probabilistic and analytic in nature. Coding theory then attempts to realize the promise of these
bounds by models which are constructed through mainly algebraic means. Different concepts
have been explained with the help of examples. A large number of problems with solutions have
been provided to assist one to get a firm grip on the ideas developed. There is a plenty of scope
for the reader to try and solve problems at his own in the form of exercises.
I am deeply indebted to all those authors whose research paper on Information
Theory and Coding influenced my learning of the subject and take this opportunity to express
my sincere gratitude to them. I am thankful to Dr. Rajesh Goel (Principal, MIET, Mohri);
Mr. R.S. Chauhan (Assistant Prof., JMIT, Radaur); Mr. Vikas Mittal (Sr. Lect. in HEC, Jagadhri)
and Mr. Amanjeet Panghal (Lect. in MIET, Mohri) for motivation and continuous encouragement
during the preparation of this manuscript. I also wish to thank my collegues and friends who
have given many valuable suggestions on the scope and contents of the book. I am also indebted
to M/s Bharat Publications, Yamuna Nagar for bringing out the book in the short time.
It is my earnest belief that no work is ever complete till it has had its share of
criticism and hence I'll be too glad to receive comments and suggestions for the betterment of
the book.
Author
(v)
4. FORWARD
It is great honour and immense pleasure for me to write a foreward of a book on
Information Theory and Coding by one of my esteemed Colleagues, Mr. Nitin Mittal.
Considering the needs of engineering students and the fact that they hardly get any
exposure to translate technology into practical applications, a basic knowledge in Information
Theory and Coding is essential and to be considered as a main subject in Electronics and
Communication Engineering. To cover the course material for such a vast and wide field, a
comprehensive and easy to understand approach to the subject is required. In this book, the
author has tried to put maximum efforts in this direction. The matter has been presented in well
structured manner, an easy to understand language which can be grasped easily by students of
different disciplines.
Attention has also been paid to the fact that the text as well as diagrams could be
reproduced by the students in theory examinations. A number of review questions given at the
end of each chapter will further enhance the understanding of basic concepts.
I am sure that this book would be quite useful to the students at undergraduate level in
various institutions, along with post graduate aspirants as well.
With my best wishes to the author.
Dr. RAJESH GOEL
Principal,
MIET, Mohri
Kurukshetra
(vi)
5. CONTENTS
Chapter 1. Probability Theory And Random Variables 1–52
1.1 Introduction 1
1.2 Probability Theory 2
1.2.1 Experiment 2
1.2.2 Sample Space And Events 2
1.2.3 Algebra of Events 3
1.3 Probability of Events 4
1.3.1 Properties of Probability 4
1.4 Conditional Probability 6
1.4.1 Conditional Probability of Independent Events 7
1.4.2 Bayes’ Formula 7
1.5 Random Variables 13
1.5.1. Discrete Random Variables 13
1.5.2. Continuous Random Variables 14
1.6 Probability Distribution of A Discrete Random Variable 14
1.7 Cumulative Distribution Function (CDF) 15
1.7.1 Properties of Cumulative Distribution Function 16
1.8 Probability Density Function (PDF) 17
1.8.1 Properties of Probability Density Function 17
1.9 Two – Dimensional Random Variables 20
1.9.1 Joint Distribution Function 20
1.9.2 Marginal Distribution Function 21
1.9.3 Independent Random Variables 21
1.9.4 Joint Probability Density Function 21
1.9.5 Marginal Probability Density Function 22
1.9.6 Conditional Probability Density Function 22
1.10 Functions of Random Variables 24
1.11 Statistical Averages of Random Variables 26
1.11.1 Expectation 26
1.11.2 Moments And Variance 27
1.11.3 Covariance And Correlation Coefficient 28
1.12 Some Important Distributions 28
1.12.1 The Uniform or Rectangular Distribution 28
1.12.2 The Exponential Distribution 29
1.12.3 Gaussian or Normal Distribution 30
1.12.4 Rayleigh Distribution 32
(vii)
6. 1.13. Characteristic Transformation Functions of Random Variables 34
1.13.1 Properties of Moment Generating Function 35
1.13.2 Determination of Statistical Averages Using MGF 36
1.14 Convergence of A Sequence of Random Variables 37
1.14.1 Law of Large Numbers 37
1.14.2 Central Limit Theorem 38
Chapter 2. Random Processes 53–86
2.1 Introduction 53
2.2. Random Processes 54
2.3 Statistical Averages of Random Process 55
2.3.1 Ensemble Averages 55
2.3.2 Time Averages 56
2.4 Stationary Random Process 57
2.4.1 Strictly Stationary Process 57
2.4.2 Wide Sense Stationary Process 58
2.5 Ergodic Process 58
2.5.1 Properties of Ergodic Random Process 59
2.6 Correlation Function 60
2.6.1 Auto-Correlation Function 61
2.6.2 Cross-Correlation Function 62
2.6.3 Auto Covariance Function 63
2.6.4 Cross Covariance Function 63
2.7 Spectral Densities 64
2.7.1 Power Spectral Density 65
2.7.2 Cross Power Spectral Density 67
2.7.3 Energy Spectral Density 67
2.8 Response of Linear Systems To The Input Random Processes 69
2.9 Special Classes of Random Processes 73
2.9.1 Gaussian Process 73
2.9.2 Markov Process 74
2.9.3 Poisson Process 75
2.9.4 White Noise or White Process 76
2.9.5. Band - Limited White Noise or Process 77
Chapter 3. Elements of Information Theory 87–134
3.1. Introduction 87
3.2. Information Sources 88
3.3. Information: A Measure of Uncertainty 88
3.4 Average Information or Entropy 89
3.4.1. Properties of Entropy 91
3.5 Binary Sources 94
3.6 Extension of A Discrete Memoryless Source 95
(viii)
7. 3.7 Measure of Information For Two - Dimensional Discrete Finite 96
Probability Scheme
3.7.1 Discrete Memoryless Channels 98
3.8 Noise Characteristics of A Channel 101
3.9 Measure of Mutual Information 102
3.9.1 Relationship Among Various Entropies 103
3.9.2 Mutual Information 103
3.9.3 Properties of Mutual Information 104
3.10 Channel Capacity 107
3.11 Channel Capacity of Binary Noise Structures 107
3.11.1 Channel Capacity of A BSC (Binary Symmetric Channel) 108
3.11.2 Channel Capacity of A BEC (Binary Erasure Channel) 109
3.12 Differential Entropy And Mutual Information For Continuous Signals 110
3.13 Shannon’s Theorem On Coding For Memoryless Noisy Channel 113
Chapter 4. Source Encoding 135–184
4.1 Introduction 135
4.2 Source Encoding 136
4.3 Basic Properties of Codes 137
4.4 Separable Binary Codes 139
4.5 Shannon – Fano Encoding 141
4.6 Noiseless Coding Theorem 144
4.7 Theorem of Decodability 149
4.8 Average Length of Encoded Messages 150
4.9 Shannon’s Binary Encoding 152
4.10 Fundamental Theorem of Discrete Noiseless Coding 154
4.11 Huffman’s Minimum – Redundancy Code 156
Chapter 5. Error Control Coding For Digital 185–206
Communication System
5.1 Introduction 185
5.2 Elements of Digital Communication System 186
5.3 Mathematical Models For Communication Channels 192
5.4 Channel Codes 194
5.5 Modulation And Coding 196
5.6 Maximum Likelihood Decoding 200
5.7 Types of Errors 202
5.8 Error Control Strategies 203
Chapter 6. Error Detection And Correction 207–224
6.1 Introduction 207
6.2 Types of Errors 208
(ix)
8. 6.3 Error Detection 209
6.3.1 Parity Check 210
6.3.2 Cyclic Redundancy Check (CRC) 211
6.3.3 Checksum 213
6.4 Error Correction 215
6.4.1 Single – Bit Error Correction 215
6.4.2. Burst Error Correction 219
Chapter 7. Field Algebra 225–254
7.1 Introduction 225
7.2 Binary Operations 225
7.3 Groups 227
7.3.1. Commutative Group 227
7.3.2. Semi – Group 228
7.3.3. Order of A Group 228
7.3.4. Addition Modulo M 228
7.3.5. Multiplication Modulo M 228
7.3.6. General Properties of Groups 230
7.4 Fields 230
7.4.1 Characteristics of The Field 234
7.5 Binary Field Arithmetic 234
7.5.1 Irreducible Polynomial Over GF (2) 236
7.5.2 Primitive Polynomial Over GF (2) 237
7.6 Construction of Galois Field GF (2m) 239
7.7 Basic Properties of Galois Field GF (2m) 243
7.8 Vector Spaces 246
7.9 Matrices 249
Chapter 8. Linear Block Codes 255–282
8.1 Introduction 255
8.2 Repetition Code 256
8.2.1 Majority Voting Decoder 256
8.2.2 Single Error Correcting Repetition Code 256
8.2.3 Advantages And Disadvantages of Repetition Codes 257
8.3 Binary Block Codes 257
8.4 Linear Block Code 258
8.4.1 Systematic Linear Block Code 259
8.4.2 Encoder For Linear Block Code 262
8.4.3 Parity – Check Matrix 263
8.5 Syndrome Calculation For Linear Block Code 264
8.5.1 Properties of The Syndrome (S) 268
8.6 The Hamming Distance of A Block Code 270
8.7 Error – Detecting And Correcting Capabilities 271
(x)
9. 8.8 Syndrome Decoding of Linear Block Code 273
8.9 Burst Error Correcting Block Codes 275
8.10 Other Important Block Codes 277
8.10.1 Hamming Codes 277
8.10.2 Extended Codes 278
Chapter 9. Cyclic Codes 283–308
9.1 Introduction 283
9.2 Cyclic Codes 284
9.3 Generator Polynomial of Cyclic Codes 285
9.4 Parity – Check Polynomial of Cyclic Codes 286
9.5 Systematic Cyclic Codes 288
9.6 Generator And Parity – Check Matrices of Cyclic Codes 290
9.7 Encoder For Cyclic Codes 292
9.8 Syndrome Polynomial Computation 295
9.9 Decoding of Cyclic Codes 297
9.10 Error – Trapping Decoding 299
9.11 Advantages And Disadvantages of Cyclic Codes 301
9.12 Important Classes of Cyclic Codes 301
Chapter 10. BCH Codes 309–338
10.1 Introduction 309
10.2 Binary BCH Codes 310
10.3 Generator – Polynomial of Binary BCH Codes 310
10.4 Parity – Check Matrix of BCH Code 314
10.5 Encoding of BCH Codes 316
10.6 Properties of BCH Codes 318
10.7 Decoding of BCH Codes 318
10.7.1 Syndrome Computation 318
10.7.2 Error Location Polynomial 320
10.8 BCH Decoder Architecture 321
10.8.1 Peterson’s Direct Algorithm 322
10.8.2 Berlekamp’s Iterative Algorithm 326
10.8.3. Chien Search Algorithm 332
10.9 Non - Primitive BCH Code 333
10.10 Non – Binary BCH Codes And RS Codes 334
Chapter 11. Convolutional Codes 339–376
11.1 Introduction 339
11.2 Convolutional Codes 340
11.3 Convolutional Encoder 341
11.3.1 Encoding Using Discrete Convolution 342
11.3.2 Encoding Using Generator Matrix 344
(xi)
10. 11.4. Structural Properties of Convolutional Codes 346
11.4.1. Code – Tree Diagram 346
11.4.2 Trellis Diagram 348
11.4.3 State Diagram Representation 349
11.5 Decoding of Convolutional Code 350
11.5.1 Maximum - Likelihood Decoding 350
11.5.2 The Viterbi Decoding Algorithm 352
11.5.3 Sequential Decoding of Convolutional Codes 356
11.6 Distance Properties of Convolutional Codes 357
11.7 Burst Error Correcting Convolutional Codes 359
Chapter 12. Basic ARQ Strategies 377–388
12.1 Introduction 377
12.2 Automatic Repeat Request (ARQ) 378
12.3 Stop-And-Wait ARQ 379
12.4 Continuous ARQ 381
12.4.1 Go-Back-N ARQ 381
12.4.2. Selective Repeat ARQ 383
12.5 Hybrid ARQ 384
Chapter 13. Cryptography 389–404
13.1 Introduction 389
13.2 Cryptography 390
13.2.1 Need of Cryptography 390
13.2.2 Cryptographic Goals 390
13.2.3 Evaluation of Information Security 391
13.3 Cryptography Components 392
13.4 Symmetric Key Cryptography 393
13.4.1 Symmetric Key encryption / decryption 394
13.4.2 Techniques for coding plain text to chiper text 394
13.4.3 Advantages and disadvantages of symmetric key cryptography 396
13.5 Asymmetric Key Cryptography 396
13.5.1 Public Key encryption / decryption 397
13.5.2 Conversion of plain text to chiper text algorithms 398
13.5.3 Advantages and disadvantages of public-key cryptography 400
13.6 Comparison between symmetric and public-key cryptography 401
13.7 Cryptography Applications 401
Sample Model Papers 405–407
References 40 8
Index 409–412
(xii)
11. REFERENCES
1. A. Bruce Carlson, Janet C. Rutledge and Crilly, “Communication Systems”, 3rd Ed., Mc
Graw Hill, 2002.
2. A. Papoulis, “Probability, Random Variables and Stochastic Processes”, Mc Graw Hill,
1991.
3. Andrew S Tanenbaum, “Computer Networks”, 3rd ed., Upper Saddle River, NJ: Prentice
Hall, 1996.
4. B. P. Lathi, “Modern Digital and Analog Communication Systems”, Third Edition, Oxford
Press, 2007.
5. B. R. Bhat, “Modern Probability Theory”, New Age International Ltd, 1998.
6. Behrouz A Forouzan, “Data Communication and Networking”, Tata McGraw-Hill, 2003.
7. D. Stinson, “Cryptography: Theory and Practice”, CRC Press, Second edition, 2000.
8. Das, Mullick and Chatterjee, “Digital Communication”, Wiley Eastern Ltd., 1998.
9. Fazlollah M. Reza, “An Introduction to Information Theory”, Dover Publications, Inc.,
New York, 1994.
10. Gregory Karpilovsky, “Field Theory: Classical Foundations and multiplicative groups”,
Tata McGraw-Hill, 1988.
11. Herbert Taub and Donald L Schilling, “Principles of Communication Systems”, 3rd Edition,
Tata McGraw Hill, 2008.
12. J. E. Pearson, “Basic communication theory”, Upper Saddle River, NJ: Prentice Hall,
1992.
13. John G. Proakis, Masoud Salehi, “Fundamentals of Communication Systems”, Pearson
Education, 2006.
14. R. E. Blahut, “Principles and Practice of Information Theory”, Addison-Wesley, Reading,
Mass, 1987.
15. R. P Singh and S. D. Sapre, “Communication Systems – Analog and Digital”, Tata McGraw
Hill, 2nd Edition, 2007.
16. R. M.Gray, L. D Davission, “Introduction to Statistical Signal Processing”, Web Edition,
1999.
17. Robert G. Gallanger, “Information Theory and Reliable Communication”, Mc Graw Hill,
1992.
18. Shu Lin and D. J. Costello, “Error Control Coding: Fundamentals and Applications”,
Prentice Hall, 1983.
19. Simon Haykin, “Communication Systems”, John Wiley & sons, New York, 4th Edition,
2001.
20. W. Stallings, “Cryptography and Network Security: Principles and Practice”, Second
edition, Prentice Hall, 1999.
21. William Stallings, “Data and Computer Communications”, 5th ed., Upper Saddle River,
NJ: Prentice Hall, 1997.
(408)