VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -2notes

1. INFORMATION THEORY AND
CODING
5th SEM E&C
JAYANTHDWIJESH H P M.tech (DECS)
Assistant Professor – Dept of E&C
B.G.S INSTITUTE OF TECHNOLOGY (B.G.S.I.T)
B.G Nagara, Nagamangala Tq, Mandya District- 571448

2. FORMULAS FOR REFERENCE
MODULE – 2 (source coding)
 Entropy of source or Average information content of the source.
H(S) = 𝑷𝒊 𝐥𝐨𝐠(
𝟏
𝑷 𝒊
𝒒
𝒊=𝟏 ) bits/symbol or H(S) = 𝑷 𝑲 𝐥𝐨𝐠 𝟐 𝟏 (
𝟏
𝑷 𝑲
𝑵
𝑲=𝟏 ) bits/symbol
 Average length
L = 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 bits/symbol or L = 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 bits/symbol
 Source or code efficiency
𝜼 𝑺=
𝑯(𝑺)
𝑳
X 𝟏𝟎𝟎% or 𝜼 𝑪=
𝑯(𝑺)
𝑳
X 𝟏𝟎𝟎%
 Source or code redundancy
𝑹 𝜼 𝑺
= 1- 𝜼 𝑺 = (1 -
𝑯(𝑺)
𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼 𝑪
= 1- 𝜼 𝑪 = (1 -
𝑯(𝑺)
𝑳
) X 𝟏𝟎𝟎%
 Compute the number of stages required for the encoding operation, which is
given by
𝒏 =
𝑵−𝒓
𝒓−𝟏
or  =
𝒒−𝒓
𝒓−𝟏
 The probability of “0”s and “1”s and “2” s in the code are found using the
formulas
P (0) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "0" s in the code for
𝒒
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] or
P (0) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "0" s in the code for𝑵
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] .
P (1) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "1" s in the code for
𝒒
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] or
P (1) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "1" s in the code for𝑵
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] .
P (2) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "2" s in the code for
𝒒
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] or
P (2) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "2" s in the code for𝑵
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] .
 The variance of the word length is calculated from
Var ( 𝒍𝒊 ) = E [( 𝒍𝒊 − 𝐋 ) 𝟐
= 𝑷𝒊
𝒒
𝒊=𝟏 ( 𝒍𝒊 − 𝐋 ) 𝟐
 The Smallest integer value of 𝒍𝒊 if found using
𝟐 𝒍 𝒊 
𝟏
𝑷𝒊
or 𝒍𝒊  𝒍𝒐𝒈 𝟐
𝟏
𝑷𝒊
 The average length 𝑳 𝟐 of the 2nd
extension is given by
𝑳 𝟐 = 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 bits/symbol or 𝑳 𝟐 = 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 bits/symbol

3.  The average length 𝑳 𝟑 of the 3rd
extension is given by
𝑳 𝟑 = 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 bits/symbol or 𝑳 𝟑 = 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 bits/symbol
 The entropy of the 2nd extended
source is calculated as
H (𝑺 𝟐
) = 2 H(S)
 The entropy of the 3rd extended
source is calculated as
H (𝑺 𝟑
) = 3H(S)
 Source or code efficiency of the 2nd
extended source is
𝜼(𝟐)
𝑺
=
𝐇 (𝑺 𝟐
)
𝑳 𝟐
X 𝟏𝟎𝟎% or 𝜼(𝟐)
𝑪
=
𝐇 (𝑺 𝟐
)
𝑳 𝟐
X 𝟏𝟎𝟎%
 Source or code redundancy of the 2nd
extended source is
𝑹(𝟐)
𝜼 𝑺
= 1- 𝜼(𝟐)
𝑺
= (1 -
𝐇 (𝑺 𝟐)
𝑳 𝟐
) X 𝟏𝟎𝟎% or 𝑹(𝟐)
𝜼 𝑪
= 𝟏 − 𝜼(𝟐)
𝑪
= (1 -
𝐇 (𝑺 𝟐)
𝑳 𝟐
)
𝐗 𝟏𝟎𝟎%
 Source or code efficiency of the 3rd
extended source is
𝜼(𝟑)
𝑺
=
𝐇 (𝑺 𝟑
)
𝑳 𝟑
X 𝟏𝟎𝟎% or 𝜼(𝟑)
𝑪
=
𝐇 (𝑺 𝟑
)
𝟑
X 𝟏𝟎𝟎%
 Source or code redundancy of the 3rd
extended source is
𝑹(𝟑)
𝜼 𝑺
= 1- 𝜼(𝟑)
𝑺
= (1 -
𝐇 (𝑺 𝟑)
𝑳 𝟐
) X 𝟏𝟎𝟎% or 𝑹(𝟑)
𝜼 𝑪
= 𝟏 − 𝜼(𝟑)
𝑪
= (1 -
𝐇 (𝑺 𝟑)
𝑳 𝟑
)
𝐗 𝟏𝟎𝟎%
 The average length 𝑳 𝟑
of the Huffman ternary code is given by
𝑳(𝟑)
= 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 trinits /Msg- symbol or 𝑳(𝟑)
= 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 trinits / Msg- symbol
 The average length 𝑳 𝟒
of the Huffman quaternary code is given by
𝑳(𝟒)
= 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 quaternary digits /Msg- symbol or
𝑳(𝟒)
= 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 quaternary digits / Msg- symbol
 The entropy in ternary units/ message symbol is found by using equation
𝐇 𝟑(S) =
𝐇(𝐒)
𝒍𝒐𝒈 𝟐 𝟑
ternary units/ message symbol or
𝐇 𝟑(S) = 𝑷 𝑲 𝐥𝐨𝐠 𝟑 𝟏 (
𝟏
𝑷 𝑲
𝑵
𝑲=𝟏 ) ternary units/ message symbol or
𝐇 𝟑(S) = 𝑷𝒊 𝐥𝐨𝐠 𝟑 𝟏 (
𝟏
𝑷 𝒊
𝒒
𝒊=𝟏 ) ternary units/ message symbol
 The entropy in quaternary units/ message symbol is found by using equation
𝐇 𝟒(S) =
𝐇(𝐒)
𝒍𝒐𝒈 𝟐 𝟒
quaternary units/ message symbol or

4. 𝐇 𝟒(S) = 𝑷 𝑲 𝐥𝐨𝐠 𝟒 𝟏 (
𝟏
𝑷 𝑲
𝑵
𝑲=𝟏 ) quaternary units/ message symbol or
𝐇 𝟒(S) = 𝑷𝒊 𝐥𝐨𝐠 𝟒 𝟏 (
𝟏
𝑷 𝒊
𝒒
𝒊=𝟏 ) quaternary units/ message symbol
 Source or code efficiency of the ternary is given by
𝜼 𝒔(𝟑)
=
𝐇 𝟑(𝐒)
𝑳(𝟑)
X 𝟏𝟎𝟎% or 𝜼 𝒄(𝟑)
=
𝐇 𝟑(𝐒)
𝑳(𝟑)
X 𝟏𝟎𝟎% or
𝜼 𝑺=
𝐇 𝟑(𝐒)
𝑳
X 𝟏𝟎𝟎% or 𝜼 𝑪=
𝐇 𝟑(𝐒)
𝑳
X 𝟏𝟎𝟎%
 Source or code efficiency of the quaternary is given by
𝜼 𝒔(𝟒)
=
𝐇 𝟒(𝐒)
𝑳(𝟒)
X 𝟏𝟎𝟎% or 𝜼 𝒄(𝟒)
=
𝐇 𝟒(𝐒)
𝑳(𝟒)
X 𝟏𝟎𝟎% or
𝜼 𝑺=
𝐇 𝟒(𝐒)
𝑳
X 𝟏𝟎𝟎% or 𝜼 𝑪=
𝐇 𝟒(𝐒)
𝑳
X 𝟏𝟎𝟎%
 Source or code redundancy of the ternary is given by
𝑹 𝜼 𝒔(𝟑)
= 1- 𝜼 𝒔(𝟑)
= (1 -
𝐇 𝟑(𝐒)
𝑳(𝟑) ) X 𝟏𝟎𝟎% or
𝑹 𝜼 𝒄(𝟑)
= 𝟏 − 𝜼 𝒄(𝟑)
= (1 -
𝐇 𝟑(𝐒)
𝑳(𝟑) ) 𝐗 𝟏𝟎𝟎% or
𝑹 𝜼 𝑺
= 1- 𝜼 𝑺 = (1 -
𝐇 𝟑(𝐒)
𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼 𝑪
= 1- 𝜼 𝑪 = (1 -
𝐇 𝟑(𝐒)
𝑳
) X 𝟏𝟎𝟎%
 Source or code redundancy of the quaternary is given by
𝑹 𝜼 𝒔(𝟒)
= 1- 𝜼 𝒔(𝟒)
= (1 -
𝐇 𝟒(𝐒)
𝑳(𝟒) ) X 𝟏𝟎𝟎% or
𝑹 𝜼 𝒄(𝟒)
= 𝟏 − 𝜼 𝒄(𝟒)
= (1 -
𝐇 𝟒(𝐒)
𝑳(𝟒) ) 𝐗 𝟏𝟎𝟎% or
𝑹 𝜼 𝑺
= 1- 𝜼 𝑺 = (1 -
𝐇 𝟒(𝐒)
𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼 𝑪
= 1- 𝜼 𝑪 = (1 -
𝐇 𝟒(𝐒)
𝑳
) X 𝟏𝟎𝟎%