02 Statistics review

Bayesian Networks
Unit 2
Statistics Review
Wang, Yuan-Kai, 王元凱
ykwang@mails.fju.edu.tw
http://www.ykwang.tw

Department of Electrical Engineering, Fu Jen Univ.
輔仁大學電機工程系

2006~2011
Reference this document as:
Wang, Yuan-Kai, “Statistics Review," Lecture Notes of Wang, Yuan-Kai,
Fu Jen University, Taiwan, 2011.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright

王元凱 Unit - Statistics Review p. 2

Goal of this Unit
 Review basic concepts of
statistics in terms of
 Image processing
 Pattern recognition



Related Units
 Previous unit(s)
 Probability Review
 Next units
 Uncertainty Inference (Discrete)
 Uncertainty Inference (Continuous)



Self-Study
 Artificial Intelligence: a modern
approach
 Russell & Norvig, 2nd, Prentice Hall,
2003. pp.462~474,
 Chapter 13, Sec. 13.1~13.3
 統計學的世界
 墨爾著，鄭惟厚譯, 天下文化，2002
 深入淺出統計學
 D. Grifiths, 楊仁和譯，2009, O’ Reilly



Contents
1. Introduction …………………………… 6
2. Histogram ……………………………… 12
3. Central Tendency .............................. 18
4. Variance ............................................. 26
5. Frequency Distribution …………...... 34
6. Covariance ......................................... 52
7. Covariance Matrix …………………… 57
8. Correlation .......................................... 64
9. Chart and Graph …………………….. 68
10.References …………………………… 79


1. Introduction
 Probability and statistics are
about uncertainty
 The world is full of uncertainty
 Our hardware/software
implementation needs to consider
uncertainty



Uncertainty by Probability
 It summarizes the uncertainty
that arises from laziness and
ignorance
 An example
 P(your toothache is caused by a
cavity) = 0.8
 20% represents your laziness
and ignorance – all other
possible causes



Uncertainty by Statistics
 It derives probabilistic facts
from a set of data
 Derive actual probability
number
 P(your toothache is caused by
a cavity) = 0.8
 Describe characteristics of data
 Mean, variance, moment, ...
 Build the statistic model of data
 Gaussian, Gaussian Mixture
 Reason new facts from the data


What Is Statistics
 Given a set of data from a random
variable
 A statistic is a number that
provides information about the
data
 Descriptive statistics
 Two way to describe data
 Measures of central tendency
 Measures of dispersion



Measures of Central
Tendency
 Mean
 Average, expected value of the
random variable
 Median
 Middle value of the R.V.
 Mode
 The variable value at the peak of the
pmf/pdf



Measures of Dispersion
 Dispersion
 Variance
 Covariance
 Correlation
 Moment
 Others: range, percentiles,
95% percentile,



2. Histogram
 This course has 15 students
 Every student has a score with
values: 0, 10, 20, ... 100
 Random variable
X = Student's score
 Scores of the 15 student
 {20, 90, 90, 100, 50, 60, 70, 60 ,80,
70, 80, 90, 80, 70, 70}
 20x1; 50x1; 60x2; 70x4; 80x3; 90x3;
100x1


The Histogram
 20x1; 50x1; 60x2; 70x4; 80x3;
90x3; 100x1 No. of X
X

10 20 30 40 50 60 70 80 90 100

 20x1/15; 50x1/15; 60x2/15;
70x4/15; 80x3/15; 90x3/15;
100x1/15 P(X)
X
Histogram is P.D.F.
10 20 30 40 50 60 70 80 90 100


Definitions
 For a random variable X
 X has n possible values
{x1, x2, ..., xn}
 Now there are N random data
of X
 x1, x2, .., xN
 Histogram & Distribution
 The number of xi : N(xi)
 The probabilities of xi :
p(xi)= N(xi)/N


Histogram v.s. P.D.F.



2D Gaussian

Histogram pdf



Histogram of an Image

• Random variable X (Gray level) has n possible
values {x1, x2, ..., xn}, n=256
• N random data x1, x2, .., xN of X, N=Width*Height
• Histogram: N(xi)
• Distribution: P(xi) = N(xi) / N


3. Central Tendency
 Random variable
X = Student's score
 Scores of the 15 student
 {20, 90, 90, 100, 50, 60, 70, 60 ,80, 70,
80, 90, 80, 70, 70}
 20x1; 50x1; 60x2; 70x4; 80x3; 90x3;
100x1
P(X)
 Histogram
X
10 20 30 40 50 60 70 80 90 100
 Mean ?


Mean
 Mean from the set of data
1 N
E[ X ]  x 
N
x
i 1
i

 Mean from the p.d.f
n
E [ X ]  x   xi p( xi )
i 1



Mean of an Image

1 N n
E[ X ]  x   xi E [ X ]  x   xi p( xi )
N i 1 i 1



Disadvantage of Mean
 Mean is easily influenced by
outlier (extreme values)
 Mean may not be the real value
P(X)
1 N
x
N
 x i  72
i 1
X
10 20 30 40 50 60 70 80 90 100



Median & Mode
 Median and mode are another
measures of central tendency
Median: (1) Sort the scores, (2) Select the middle
{20, 50, 60, 60, 70, 70, 70, 70, 80 ,80, 80, 90, 90, 90, 100}

Mode: select the score with the maximum N(X) or P(X)
20x1; 50x1; 60x2; 70x4; 80x3; 90x3; 100x1
P(X)
X
10 20 30 40 50 60 70 80 90 100



Advantage of Median & Mode
 Median and mode is not
influenced by outlier
 Median and mode will be the real
value



Expected Value
 E[X] : mean
n
E[ X ]  x   xi p ( xi ) E[ X ]  x 


i 1
 xp( x)dx 

 E[Xr] – rth moment of X
n 
E[ X ]   xi p ( xi ) E[ X ]   x r p ( x)dx
r r

i 1
 E[(X-µ)r] – rthn central moment of X
E[( X   ) ]   ( xi   ) p( xi )
r r

i 1

E[( X   ) ]   ( x   ) p ( x)dx
r r



Deviation about the Mean

x xi

 It indicates how far a value is
from the center
 It is a very important number to
measure the dispersion of how a
distribution spreads out



4. Variance
 Variance and standard deviation
come from the “deviation”
 Average Deviation
 Calculate all of the deviations and
find their average
 It is a measure of the typical amount
any given data point might vary
N

 ( xi  x )
x x
i
AD  i 1
N


We need Absolute Deviation

xi x x
i
N xi | xi  x |
1 1-3=-2  xi  x 1 |1-3|=2
2 2-3=-1 AAD  i 1 2 |2-3|=1
3 3-3=0 N 3 |3-3|=0
4 4-3=1 4 |4-3|=1
5 5-3=2 5 |5-3|=2
=15 =? N =15 =6
x  15/5  (x i
 x)
x  15/5 ABD=6/
AD  i 1
=3.0 N =3.0 5 =1.2


Or Square of the Deviation
N
Square the
deviations  ( x i  x ) 2 Take the
square root
to remove Variance  i 1
to return to the
minus signs N
original scale
N

 x xi N

 (x i
 x) 2

AAD  i 1
N  i 1
N
N

 (x i
 x)
AD  i 1
N


Sample Mean
and Sample Variance
 We can approximate the expected
value by the sample mean
N
x 1
N x
i 1
i

N
 Sample variance s 
i
2 1
N  (x  x)
i
2

But, strangely enough, if you1want a good
approximation of the true variance, you
should use 2 N 2 1 N i
 
ˆ
N 1
s   (x  x)
N  1 i 1
2



Variance of an Image

n 1
1  2   ( xi  x ) 2 p ( xi ), n  256
N
 
ˆ2

N  1 i 1
( xi  x )2 ˆ
i 0
1 n
x
N
 x p( x ) Moments
i 1
i i



An Example of Variance
 Variance of the scores of 15
students in this course = ?
 {20, 90, 90, 100, 50, 60, 70, 60 ,80, 70,
80, 90, 80, 70, 70}
P(X)
1 N

 x   x  72
i
X
N i 1 10 20 30 40 50 60 70 80 90 100
1 N i
2 
ˆ 
N  1 i 1
( x  x )2

= 388.6


Ex. of Standard Deviation
 Standard deviation (SD): 
 = (Var)1/2
1 N

ˆ  ( x  x ) = 19.7
N  1 i 1
i 2

P(X)
1 N

| x
X
10 20 30 40 50 60 70 80 90 100  i
x|
N i 1

52.3 72 91.7


Formal Definition of Variance

Var ( X )   x  E[( X  E[ X ]) 2 ]   ( xi  x ) 2 p( xi )
2

i

Var ( X )    E[( X  E[ X ]) ]   ( xi  x ) p( xi )dx
2
x
2 2

x



5. Frequency Distributions
• A graph or chart that shows the
number of observations of a
given value, or class interval



Shape of Distribution
 Modality
 The number of peaks in the curve
 Skewness
 An asymmetry in a distribution
where values are shifted to one
extreme or the other.
 Kurtosis
 The degree of Peakedness/flatness
in the curve



Modality
 Unimodal

 Bimodal

 Multimodal



Skewness (1/3)
 The third moment about the
Mean
 =0: symmetry distribution
(Normal distribution )
 >0 : Right Skew (Positive Skew)
 <0 : Left Skew (Negative Skew)

Right Left
skew Symmetry skew



Skewness (2/3)

 x 
 Skewness formula N
3
from data
i
x
i 1
N  1
 Skewness formula
from p.d.f n

 x  x 
3
i p ( xi )
i 1



Skewness (3/3)
Coefficient of Skewness
(Normalized skewness)

 x  x 
N
 Skewness formula i 3
from data i 1
3
N  1
n

 x  x 
 Skewness formula 3
p ( xi )
from p.d.f i 1
i

3



Kurtosis (1/3)
 The normalized fourth
moment about the Mean
 K=3: normal peak (Gaussian)
 K>3: sharp peak
 K<3: flat peak
K=3



Kurtosis (2/3)
 x 
N
4
 From data
i
x
i 1
4
N  1
n

 x  x 
4
 From p.d.f i p ( xi )
i 1
4




Kurtosis (3/3)
 Another definition of kurtosis
 K=0: normal peak (Gaussian)
 K>0: sharp peak
 K<0: flat peak

 x 
N n

 x  x 
4
i
x i
4
p ( xi )
4 3
i 1
N  1
i 1
4 3




Demo
 A demo of shape of frequency
distribution



Shape of Distribution
of an Image



Frequency Distributions -
Types
 The Normal
 The Uniform
 The Log-normal
 The Exponential
 Statistical Distributions
 t
 -Square
 F



Frequency Distributions –
Types (cont.)
 Hyper-geometric
 Poisson
 Binomial
 Gamma
 Weibull
 Cauchy
 Beta



Gaussian Dist. Of Data (1/5)

Observe: Mean, Principal axes,
implication of off-diagonal Common convention: show contour
covariance term, max gradient corresponding to 2 standard
zone of p(x) deviations from mean




In this example, x and y are almost
independent



In this example, x and “x+y” are clearly
not independent



In this example, x and “20x+y” are
clearly not independent


6. Covariance
 In addition to know the mean &
variance of random variables X & Y
 We usually want to know
 If X increase,
 Will Y probably increase or
decrease?
 We want to know the probable
relationship between X and Y
 We will use covariance to identify the
relationship


Mean Vector
 Random variable X: score of
student of this course
 Random variable Y: score of
student of another course
 Z=(X,Y) is called a random vector
 Mean vector ( x , y )is the mean of Z
Ny
1 Nx
1
x
Nx
x i
y
Ny
y
i 1
i

i 1



Variance Vector ?
 Variance of X and Y: x2, y2
1 Nx
  VAR[ X ] 
2
x
N x  1 i 1
 ( x i  x )2
Ny
1
  VAR[Y ] 
2
y 
N y  1 i 1
( y i  y )2

Variance vector (x2, y2)? No. Not Enough.
•We want to know
•If X increase,
•Will Y probably increase or decrease?


Covariance
 For two random variables X and Y
 Covariance indicates the
tendency of the two random
variables X & Y to vary together
 i.e., to co-vary
Cov ( X , Y )   XY  E[( X  E[ X ])(Y  E[Y ])]
  ( xi  x )( yi  y ) p( xi , yi )
i
1 N i
 
N  1 i 1
( x  x )( y i  y )



Property of Covariance
 If X and Y tend to increase
together, then XY>0
 If X tends to decrease when Y
increases, then XY<0
 If X and Y are uncorrelated, then
 XY=0
 |XY|≤ XY, where X is the
standard deviation of X
 XX = 2X = VAR(X)
 XY = YX


7. Covariance Matrix
 For two random variables X, Y
 There are two variances XX, YY
and two covariance XY ,YX
 We usually write the four
variances into a covariance
matrix   E[(x  μ)(x  μ)T ]
 E[( x μx ) ]
2
E[( x μx )( y μy )]
 
 E[( x μx )( y μy )] E[( x μx ) ] 
2
 
  x  xy 
2

 2 
 xy  y 
 


Quiz



Quiz
 For 3 random variables: x,y,z
 The random vector v=(x,y,z)
 The covariance matrix of v?



Independence and
Covariance



Formal Definition (1/2)
 For a random vector
X = (X1, X2, ..., Xm) with n random
variables
 The probability of x =(x1, x2, ..., xn),
P(x), is a joint probability
P(x) = P(x1, x2, ..., xn)
 Its expected values are
a mean vector  = (1, 2, ..., m)
 E[X] = (E[X1], E[X2], ..., E[Xm])
= (1, 2, ..., m) = 


Formal Definition (2/2)
 For the random vector
X = (X1, X2, ..., Xm)
 All of its covariance values are
a covariance matrix
Cov (X)  E[(X   )(X   )T ]
 E[( X 1  1 )( X 1  1 )]  E[( X 1  1 )( X m   m )] 
    

 E[( X m   m )( X 1  1 )]  E[( X m   m )( X m   m )]
 
  12  12   1m  •  is a square and
 
 21  2   2 m 
2
 symmetric matrix
      • Its diagonal elements
 2 
 m1  m 2   m 
  are 2i


Property of Covariance Matrix

 Symmetric matrix
 Positive definition matrix
 Eigenvalue



8. Correlation
 Correlation is another measure of
the relationship of two random
variables Cov( X , Y )  XY
 XY  
Var ( X )Var (Y )  X  Y
• -1  XY  1
• XY = 0 means that the variables are
uncorrelated
• XY = 0 E[XY] = E[X] E[Y]
• Independence means uncorrelated
• But not vice versa!


Correlation & Covariance
 |XY|≤ XY
 -XY ≤ XY ≤ XY
 XY = XYXY
 -1  XY  1
 XY is called the correlation
coefficient 
-1   xy 
xy
 +1
 x  y


Covariance v.s. Correlation
Y
Y

X X
XY = XY XY = 1/2XY
XY = +1 XY = 1/2
Y Y Y

X X X
XY = -XY XY = -1/2XY XY = 0
XY = -1 XY = -1/2 XY = 0


Demos
 Flash demos of correlation with
scatter plot



9. Charts and Graphs
 Descriptive Graphs
 Bar Chart
 Pie Graph
 Line Graph
 Distributions
 Histogram
 Box Plot
 Steam and Leaf



Bar Charts
 Best for displaying actual values.
 Can handle moderate # of cases
(bars)
 Excel calls it a column chart



Bar Chart – An Example
Vote for President in 2000

60
Millions

50
40
30
20
10
0
e

r
sh
an

e
or

ad
Bu
an

G

N
ch
Bu



Pie Charts
 Best used with small number of
categories or cases to display
 Good for showing relative
distribution
 Percentages, proportions
 Use only one column of data
 Plus one column of labels



Pie Chart Examples

Bush
Nader
Kerry


Buchanan
Gore
Bush
Nader



Line graphs
 Best for showing data across
time
 Always give dates
 Label X axis
 Indicate units on Y axis
 Use legend for multiple lines



Line Graphs - Example
Defense Spending - 1940-2002

400000

300000
Millions $

Defense
200000
Spending
100000

0
1940
1948
1956
1964
1972
1980
1988
1996
Year



Box Plot
 Quick picture of a distribution
 Parts of box give distribution
characteristics
 Your Text is not quite accurate!



Stem and Leaf Plot
 Good for showing distribution
while preserving data
 Figuring out stems can be tricky



Review for Next Unit
 We learn
 Histogram, mean, variance,
covariance, correlation
 Frequency distribution, Gaussian
 We will learn in next unit
 Moment, Gaussian Mixture, Linear
Gaussian
 Moment  mean, variance



Mean & Variance of an Image

1 n
2 
ˆ
1 N i
 ( x  x )2
2 
ˆ 
N  1 i 1
( xi  x ) 2 p ( xi )
N  1 i 1
1 n
x
N
 x p( x ) Moments
i 1
i i



10. References
 An excellent textbook
統計學的世界，鄭惟厚譯，天下文化2002
 Statistics: concepts and controversies,
5th, D. S. Moore, 2001
 Chart & graph: Chap. 10,11
 Central tendency & dispersion: Chap. 12
 Gaussian distribution: Chap. 13
 Correlation: Chap. 14, 15



透過Excel學統計
 統計學，劉明德等著，全華，93
 Chart & graph : Chap. 2
 Central tendency & dispersion: Chap. 3
 Frequency distribution: Chap. 6,7
 統計學與Excel，王文中著，博碩，93
 Central tendency (mean, mode, median) :
Chap. 2
 Dispersion (variance, standard deviation) :
Chap. 3
 Gaussian distribution : Chap. 4
 Correlation: Chap. 11


02 Statistics review

Recomendados

Recomendados

Más contenido relacionado

Similar a 02 Statistics review

Similar a 02 Statistics review (17)

Más de IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing

Más de IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing (9)

Último

Último (20)

02 Statistics review