Se está descargando tu SlideShare. ×

Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Cargando en…3
×

1 de 122 Anuncio

# Lecture 1

Descriptive analytics

Descriptive analytics

Anuncio
Anuncio

Anuncio

Anuncio

### Lecture 1

1. 1. Some important information on the course Lecturer: Jorge M. Mendes (jmm@novaims.unl.pt) Lectures timetable: Tuesdays and Thursdays, 3:30pm to 4:15pm Assessment rules: check syllabus Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 1 / 122
2. 2. Some important information on the course Textbooks Everitt, B. and Hothorn, T. (2011). An Introduction to Applied Multivariate Analysis with R, Springer. Johnson, R.A and Wichern (2007). D. W., Applied Multivariate Statistical Analysis, 6th edition, Pearson Prentice Hall. online tools and references: https://www.datacamp.com online tools and references: http://www.statmethods.net And... Reis, E. (1997), Estat´ıstica Multivariada Aplicada, Edi¸c˜oes S´ılabo Sharma, S. (1996). Applied Multivariate Techniques, John Wiley & Sons Timm, N. H. (2002). Applied Multivariate Analysis, Springer Other things: lecture slides, data ﬁles, etc. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 2 / 122
3. 3. Some important information on the course Course contents Data analysis basics: organisation of multivariate data; parameters and descriptive statistics; matrix manipulations for sample statistics; review of matrix deﬁnitions and operations R tutorial; fundamentals on data manipulation on R Graphical display of multivariate data; Linear combinations of random variables The multivariate normal distribution Principal components analysis Factor analysis Discriminant analysis Cluster analysis Canonical correlation analysis Multidimensional scaling ... eventually something else! Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 3 / 122
4. 4. Lecture 1 Multivariate Data Analysis Basics Jorge M. Mendes Nova Information Management School Universidade Nova de Lisboa 2015 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 4 / 122
5. 5. First things ﬁrst This ﬁrst lesson will introduce the basic tools to deal with multivariate data and multivariate techniques. Topics covered include multivariate sample statistics and spectral decomposition of a matrix. After successfully completing this lesson, you should be able to: Understand the notation used in multivariate analysis; Understand the organization of multivariate data using vector and matrix notation; Perform basic operations on matrices and vectors; Interpret measures of central tendency, dispersion, and association; Calculate sample means, variances, covariances, and correlations using a hand calculator; Compute the spectral decomposition of a matrix (eigenvalues, eigenvectors, etc.) Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 5 / 122
6. 6. Multivariate data analysis basics Organisation of multivariate data Subsection 1 Organisation of multivariate data Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 6 / 122
7. 7. Multivariate data analysis basics Organisation of multivariate data Introduction Univariate statistics is concerned with random scalar variable Y . In multivariate analysis, we are concerned with the joint analysis of multiple dependent variables. These variables can be represented using matrices and vectors. This provides simpliﬁcation of notation and a format for expressing important formulas. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 7 / 122
8. 8. Multivariate data analysis basics Organisation of multivariate data Example Suppose that we measure the variables x1 = height (cm), x2 = left forearm length (cm) and x3 = left foot length for participants in a study of the physical characteristics of adult humans. These three variables can be represented in the following column vector: x =   x1 x2 x3   Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 8 / 122
9. 9. Multivariate data analysis basics Organisation of multivariate data Example The observed data for a speciﬁc individual, say the i-th individual, might also be represented in an analogous vector. Suppose that the i-th person in the sample has height = 175 cm, forearm length = 25.5 cm and foot length = 27 cm. In vector notation these observed data could be written as: xi =   xi1 xi2 xi3   =   175 25.5 27.0   Notice the use and placement of the subscript i to represent the i-th individual. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 9 / 122
10. 10. Multivariate data analysis basics Organisation of multivariate data Deﬁnitions of matrix and vector A matrix is two-dimensional array of numbers of formulas. A vector is a matrix with either only one column or only one row. A column vector has only one column. A row vector has only one row. The dimension of a matrix is expressed as number of rows × number of columns. For instance, a matrix with 10 rows and 3 columns is said to be a 10 × 3 matrix. The vectors written in previous Example are 3 × 1 matrices. A square matrix is one for which the numbers of rows and columns are the same. For instance, a 4 × 4 matrix is a square matrix. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 10 / 122
11. 11. Multivariate data analysis basics Organisation of multivariate data The data matrix in multivariate problems Usually the observed data are represented by a matrix in which the rows are observations and the columns are variables. This is exactly the way the data are normally prepared for statistical software. The usual notation is n: the number of observed units (people, animals, companies, etc.) p: number of variables measured on each unit. Thus the data matrix will be an n × p matrix. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 11 / 122
12. 12. Multivariate data analysis basics Organisation of multivariate data Example Suppose that we have scores for n = 6 college students who have taken the verbal and the science subtests of a College Qualiﬁcation test (CQT). We have p = 2 variables: (1) the verbal score and (2) the science score for each student. The data matrix is the following 6 × 2 matrix: X =         41 26 39 26 53 21 67 33 61 27 67 29         Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 12 / 122
13. 13. Multivariate data analysis basics Organisation of multivariate data Example In the matrix just given, the ﬁrst column gives the data for x1 = verbal score whereas the second column gives data for x2 = science score. Each row gives data for a student in the sample. To repeat - the rows are observations, the columns are variables. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 13 / 122
14. 14. Multivariate data analysis basics Organisation of multivariate data Notation notes Note that we have used a small x to denote the vector of variables in ﬁrst example and a large X to represent the data matrix in second example. It should also be noted that, in matrix terms, the i-th row in the data matrix X is the transpose of the data vector xi = xi1 xi2 , as we deﬁned data vectors in ﬁrst Example. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 14 / 122
15. 15. Multivariate data analysis basics Parameters and descriptive statistics Subsection 2 Parameters and descriptive statistics Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 15 / 122
16. 16. Multivariate data analysis basics Parameters and descriptive statistics Introduction Three aspects of the data are of importance, the ﬁrst two of which you should already be familiar with from univariate statistics. These are: Central Tendency. What is a typical value for each variable? Dispersion. How far apart are the individual observations from a central value for a given variable? Association. This might (or might not!) be a new measure for you. When more than one variable are studied together how does each variable relate to the remaining variables? How are the variables simultaneously related to one another? Are they positively or negatively related? Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 16 / 122
17. 17. Multivariate data analysis basics Parameters and descriptive statistics Population parameters and sample statistics Statistics, as a subject matter, is the science and art of using sample information to make generalizations about populations. A population is the collection of all people, plants, animals, or objects of interest about which we wish to make statistical inferences (generalizations). A population parameter is a numerical characteristic of a population. In nearly all statistical problems we do not know the value of a parameter because we do not measure the entire population. We use sample data to make an inference about the value of a parameter. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 17 / 122
18. 18. Multivariate data analysis basics Parameters and descriptive statistics Population parameters and sample statistics A sample is the subset of the population that we actually measure or observe. A sample statistic is a numerical characteristic of a sample. A sample statistic estimates the unknown value of a population parameter. Information collected from sample statistic is sometimes referred to as descriptive statistic. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 18 / 122
19. 19. Multivariate data analysis basics Parameters and descriptive statistics Notation Here are the notation that will be used: Xij : Observation for variable j in subject i. p: Number of variables n: Number of subjects Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 19 / 122
20. 20. Multivariate data analysis basics Parameters and descriptive statistics Example: USDA Women’s Health Survey Let us take a look at an example. In 1985, the USDA commissioned a study of women’s nutrition. Nutrient intake was measured for a random sample of 737 women aged 25-50 years. The following variables were measured: Calcium (mg) Iron (mg) Protein (g) Vitamin A (µg) Vitamin C (mg) Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 20 / 122
21. 21. Multivariate data analysis basics Parameters and descriptive statistics Example: USDA Women’s Health Survey As a notation example let’s use the women’s nutrition data. p = 5 nutritional outcomes n = 737 subjects Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 21 / 122
22. 22. Multivariate data analysis basics Parameters and descriptive statistics Example: USDA Women’s Health Survey In multivariate statistics we will always be working with vectors of observations. So in this case we are going to arrange the data for the p variables on each subject into a vector. In the expression below, Xi is the vector of observations for the i-th subject, i = 1, ..., n(= 737). Therefore, the data for the j-th variable will be located in the j-th element of this subject’s vector, j = 1, ..., p(= 5). Xi =      Xi1 Xi2 ... Xip      Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 22 / 122
23. 23. Multivariate data analysis basics Parameters and descriptive statistics The mean Throughout this course, we’ll use the ordinary notations for the mean of a variable. That is, the symbol µ is used to represent a (theoretical) population mean and the symbol ¯x is used to represent a sample mean computed from observed data. In the multivariate setting, we add subscripts to these symbols to indicate the speciﬁc variable for which the mean is being given. For instance, µ1 represents the population mean for variable x1 and ¯x1 denotes a sample mean based on observed data for variable x1. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 23 / 122
24. 24. Multivariate data analysis basics Parameters and descriptive statistics The mean The population mean is the measure of central tendency for the population. Here, the population mean for variable j is µj = E(Xij ) The notation E stands for statistical expectation; here E(Xij ) is the mean of Xij over all members of the population, or equivalently, over all random draws from a stochastic model. For example, µj = E(Xij ) may be the mean of a normal variable. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 24 / 122
25. 25. Multivariate data analysis basics Parameters and descriptive statistics The mean The population mean µj for variable j can be estimated by the sample mean ¯xj = 1 n n i=1 Xij Note: the sample mean ¯xj , because it is a function of our random data is also going to have a mean itself. In fact, the population mean of the sample mean is equal to population mean µj ; i.e., E(¯xj ) = µj Therefore, the ¯xj is unbiased for µj . Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 25 / 122
26. 26. Multivariate data analysis basics Parameters and descriptive statistics The mean Another way of saying this is that the mean of the ¯xj ’s over all possible samples of size n is equal to µj . Recall that the population mean vector is µ which is a collection of the means for each of the population means for each of the diﬀerent variables. µ =      µ1 µ2 ... µp      Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 26 / 122
27. 27. Multivariate data analysis basics Parameters and descriptive statistics The mean We can estimate this population mean vector, µ, by ¯x. This is obtained by collecting the sample means from each of the variables in a single vector. This is shown below. ¯x =      ¯x1 ¯x2 ... ¯xp      =      1 n n i=1 Xi1 1 n n i=1 Xi2 ... 1 n n i=1 Xip      = 1 n n i=1 Xi Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 27 / 122
28. 28. Multivariate data analysis basics Parameters and descriptive statistics The mean Just as the sample means, ¯xj , for the individual variables are unbiased for their respective population means, note that the sample mean vector is unbiased for the population mean vector. E(¯x) = E      ¯x1 ¯x2 ... ¯xp      =      E(¯x1) E(¯x2) ... E(¯xp)      =      µ1 µ2 ... µp      = µ Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 28 / 122
29. 29. Multivariate data analysis basics Parameters and descriptive statistics Dispersion: variance and standard deviation A variance measures the degree of spread (dispersion) in a variable’s values. Theoretically, a population variance is the average squared diﬀerence between a variable’s values and the mean for that variable. The population variance for variable xj is σ2 j = E(xj − µj )2 Note that the squared residual (xj − µj )2 is a function of the random variable Xij . Therefore, the squared residual itself is random, and has a population mean. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 29 / 122
30. 30. Multivariate data analysis basics Parameters and descriptive statistics Dispersion: variance and standard deviation The population variance is thus the population mean of the squared residual. We see that if the data tend to be far away from the mean, the squared residual will tend to be large, and hence the population variance will also be large. Conversely, if the the data tend to be close to the mean, the squared residual will tend to be small, and hence the population variance will also be small. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 30 / 122
31. 31. Multivariate data analysis basics Parameters and descriptive statistics Dispersion: variance and standard deviation The population variance σ2 j can be estimated by the sample variance s2 j = 1 n − 1 n i=1 (Xij − ¯xj )2 = 1 n − 1 n i=1 X2 ij − n n − 1 ¯x2 j The ﬁrst expression in this formula is most suitable for interpreting the sample variance. We see that it is a function of the squared residuals; that is, take diﬀerence between the individual observations and their sample mean, and then square the result. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 31 / 122
32. 32. Multivariate data analysis basics Parameters and descriptive statistics Dispersion: variance and standard deviation Here, we may observe that if tend to be far away from their sample means, then the squared residuals and hence the sample variance will also tend to be large. If on the other hand, if the observations tend to be close to their respective sample means, then the squared diﬀerences between the data and their means will be small, resulting is a small sample variance value for that variable. The last part of the expression above, gives the formula that is most suitable for computation, either by hand or by a computer! Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 32 / 122
33. 33. Multivariate data analysis basics Parameters and descriptive statistics Dispersion: variance and standard deviation Since the sample variance is a function of the random data, the sample variance itself is a random quantity, and so has a population mean. In fact, the population mean of the sample variance is equal to the population variance: E(s2 j ) = σ2 j That is, the sample variance s2 j is unbiased for the population variance σ2 j . Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 33 / 122
34. 34. Multivariate data analysis basics Parameters and descriptive statistics Dispersion: variance and standard deviation Some textbooks use a sample variance formula derived using maximum likelihood estimation principles. In this formula, the division is by n rather than n − 1. s2 j = n i=1(xij − ¯xj )2 n Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 34 / 122
35. 35. Multivariate data analysis basics Parameters and descriptive statistics Example Suppose that we have observed the following n = 5 resting pulse rates: 64, 68, 74, 76, 78. The sample mean is ¯x = 64+68+74+76+78 5 = 72. The maximum likelihood estimate of the variance is s2 = (64 − 72)2 + (68 − 72)2 + (74 − 72)2 + 5 +(76 − 72)2 + (78 − 72)2 5 = 136 5 = 27.2 The standard deviation based in this method is s = √ 27.2 = 5.215. The more commonly used variance estimate, the one given by statistical software, would be 136 5−1 = 34. The standard deviation would be s = √ 34 = 5.83. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 35 / 122
36. 36. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation Association is concerned with how each variable is related to the other variable(s). In this case the ﬁrst measure that we will consider is the covariance between two variables j and k. The population covariance is a measure of the association between pairs of variables in a population. Here, the population covariance between variables j and k is σjk = E{(Xij − µj )(Xik − µk )} Note that the product of the residuals (Xij − µj ) and (Xik − µk ) for variables j and k, respectively, is a function of the random variables Xij and Xik . Therefore, (Xij − µj )(Xik − µk ) is itself random, and has a population mean. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 36 / 122
37. 37. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation The population covariance is deﬁned to be the population mean of this product of residuals. We see that if either both variables are greater than their respective means, or if they are both less than their respective means, then the product of the residuals will be positive. Thus, if the value of variable j tends to be greater than its mean when the value of variable k is larger than its mean, and if the value of variable j tends to be less than its mean when the value of variable k is smaller than its mean, then the covariance will be positive. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 37 / 122
38. 38. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation Positive population covariances mean that the two variables are positively associated; variable j tends to increase with increasing values of variable k and vice-versa. Negative association can also occur. If one variable tends to be greater than its mean when the other variable is less than its mean, the product of the residuals will be negative, and you will obtain a negative population covariance. Variable j will tend to decrease with increasing values of variable k. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 38 / 122
39. 39. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation The population covariance σjk between variables j and k can be estimated by the sample covariance. This can be calculated using the formula below: sjk = 1 n − 1 n i=1 (Xij − ¯xj )(Xik − ¯xk ) = 1 n − 1 n i=1 Xij Xik − n n − 1 ¯x2 j ¯x2 k Just like in the formula for variance we have two expressions that make up this formula. The ﬁrst half of the formula is most suitable for understanding the interpretation of the sample covariance, and the second half of the formula is what is used for calculation. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 39 / 122
40. 40. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation Looking at the ﬁrst half of the expression what you see inside the sum is the product of the residual diﬀerences between variable j and its mean times the residual diﬀerences between variable k and its mean. We can see that if either both variables tend to be greater than their respective means or less than their respective means, then the product of the residuals will tend to be positive leading to a positive sample covariance. Conversely if one variable takes values that are greater than its mean when the opposite variable takes a value less than its mean, then the product will take a negative value. In the end, when you add up this product over all of the observations, you will end up with a negative covariance. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 40 / 122
41. 41. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation So, in eﬀect, a positive covariance would indicate a positive association between the variables j and k. And a negative association is when the covariance is negative. For computational purposes we will use the second half of the formula. For each subject, the product of the two variables is obtained, and then the products are summed to obtain the ﬁrst term in the numerator. The second term in the numerator is obtained by taking the product of the sums of variable over the n subjects, then dividing the results by the sample size n. The diﬀerence between the ﬁrst and second terms is then divided by n − 1 to obtain the covariance value. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 41 / 122
42. 42. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation Again, sample covariance is a function of the random data, and hence, is random itself. As before, the population mean of the sample covariance sjk is equal the population covariance σjk ; i.e., E(sjk ) = σjk That is, the sample covariance sjk is unbiased for the population covariance σjk . Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 42 / 122
43. 43. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation The sample covariance is a measure of the association between a pair of variables: sjk = 0 implies that the two variables are uncorrelated. (Note that this does not necessarily imply independence, we’ll get back to this later.) sjk > 0 implies that the two variables are positively correlated; i.e., values of variable j tend to increase with increasing values of variable k. The larger the covariance, the stronger the positive association between the two variables. sjk < 0 implies that the two variables are negatively correlated; i.e., values of variable j tend to decrease with increasing values of variable k. The smaller the covariance, the stronger the negative association between the two variables. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 43 / 122
44. 44. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation Recall, that we had collected all of the population means of the p variables into a mean vector. Likewise, the population variances and covariances can be collected into the population variance-covariance matrix. This is also known by the name of population dispersion matrix. Σ =      σ2 1 σ12 . . . σ1p σ21 σ2 2 . . . σ2p ... ... ... ... σp1 σp2 . . . σ2 p      Note that the population variances appear along the diagonal of this matrix, and the covariance appear in the oﬀ-diagonal elements. So, the covariance between variables j and k will appear in row j and column k of this matrix. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 44 / 122
45. 45. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation The population variance-covariance matrix may be estimated by the sample variance-covariance matrix. The population variances and covariances in the above population variance-covariance matrix are replaced by the corresponding sample variances and covariances to obtain the sample variance-covariance matrix: S =      s2 1 s12 . . . s1p s21 s2 2 . . . s2p ... ... ... ... sp1 sp2 . . . s2 p      Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 45 / 122
46. 46. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation Note that the sample variances appear along diagonal of this matrix and the covariances appear in the oﬀ-diagonal elements. So the covariance between variables j and k will appear in the jk-th element of this matrix. Notes: S (the sample variance-covariance matrix) is symmetric; i.e., sjk = skj . S is unbiased for the population variance covariance matrix Σ ; i.e., E(S) =      E(s2 1 ) E(s12) . . . E(s1p) E(s21) E(s2 2 ) . . . E(s2p) ... ... ... ... E(sp1) E(sp2) . . . E(s2 p )      = Σ Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 46 / 122
47. 47. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation Since this matrix is a function of our random data, this means that the elements of this matrix are also going to be random, and the matrix on the whole is random as well. The statement “σ2 is unbiased”means that the mean of each element of that matrix is equal to the corresponding elements of the population. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 47 / 122
48. 48. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation In matrix notation sample variance-covariance matrix may be computed used the following expressions: S = 1 n − 1 n i=1 (Xi − ¯x)(Xi − ¯x) = 1 n − 1 n i=1 Xi Xi − n n − 1 ¯x¯x Just as we have seen in the previous formulas, the ﬁrst half of the formula is used in interpretation, and the second half of the formula is what is used for calculation purposes. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 48 / 122
49. 49. Multivariate data analysis basics Parameters and descriptive statistics Measures of association: covariance and correlation Looking at the second term you can see that the ﬁrst term in the numerator involves taking the data vector for each subject and multiplying by its transpose. The resulting matrices are then added over the n subjects. To obtain the second term in numerator, ﬁrst compute the sum of the data vectors over the n subjects, then take the resulting vector and multiply by its transpose; then divide the resulting matrix by the squared number of subjects n2 . Take the diﬀerence between the two terms in the numerator and divide by n − 1. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 49 / 122
50. 50. Multivariate data analysis basics Parameters and descriptive statistics Example Suppose that we have observed the following n = 4 observations for variables x1 and x2. x1 x2 6 3 10 4 12 7 12 6 The sample means are x1 = 10 and x2 = 5. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 50 / 122
51. 51. Multivariate data analysis basics Parameters and descriptive statistics Example The maximum likelihood estimate of the covariance is the average product of deviations from the mean: s12 = (6 − 10)(3 − 5) + (10 − 10)(4 − 5)+ 4 +(12 − 10)(7 − 5) + (12 − 10)(6 − 5) 4 = 8 + 0 + 4 + 2 4 = 3.5 The positive value reﬂects the fact that as x1 increases, x2 also tends to increase. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 51 / 122
52. 52. Multivariate data analysis basics Parameters and descriptive statistics Problem Problem: The magnitude of the covariance value is not particularly helpful as it is a function of the magnitudes (scales) of the two variables. It is hard to distinguish the eﬀects of the association between the two variables from the eﬀects of their dispersions. Note, however, that the covariance between variables j and k must lie between the product of the two component standard deviations of variables j and k, and negative of that same product: −sj sk ≤ sjk ≤ sj sk . Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 52 / 122
53. 53. Multivariate data analysis basics Parameters and descriptive statistics Example In an undergraduate statistics class, n = 30 females reported their heights (inches), and also measured their left forearm length (cm), left foot length (cm), and head circumference (cm). The sample variance-covariance matrix is the following: Notice that the matrix has four rows and four columns because there are four variables being considered. Also notice that the matrix is symmetric. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 53 / 122
54. 54. Multivariate data analysis basics Parameters and descriptive statistics Example Here are a few examples of the information in the matrix: The variance of the height variable is 8.74. Thus the standard deviation is√ 8.74 = 2.956. The variance of the left foot measurement is 1.908 (in the 3rd diagonal element). Thus the standard deviation for this variable is √ 1.908 = 1.381. The covariance between height and left arm is 3.022, found in the 1st row, 2nd column and also in the 2nd row, 1st column. The covariance between left foot and left arm is 1.234, found in the 3rd row, 2nd column and also in the 2nd row, 3rd column. All covariance values are positive so all pairwise associations are positive. But, the magnitudes do not tell us about the strength of the associations. To assess the strength of an association, we use correlation values. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 54 / 122
55. 55. Multivariate data analysis basics Parameters and descriptive statistics Correlation This suggests an alternative measure of association. The population correlation is deﬁned to be equal to the population covariance divided by the product of the population standard deviations: ρjk = σjk σj σk . The population correlation may be estimated by substituting into the formula the sample covariances and standard deviations: rjk = sij si sj = n i=1 Xij Xik − ( n i=1 Xij )( n i=1 Xik )/n2 ( n i=1 X2 ij − ( n i=1 Xij /n)2)( n i=1 X2 ik − ( n i=1 Xik /n)2) Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 55 / 122
56. 56. Multivariate data analysis basics Parameters and descriptive statistics Correlation It is very important to note that the population as well as the sample correlation must lie between -1 and 1. −1 ≤ ρjk ≤ 1 −1 ≤ rjk ≤ 1 Therefore: ρjk = 0 indicates, as you might expect, that the two variables are uncorrelated . ρjk close to +1 will indicate a strong positive dependence ρjk close to -1 indicates a strong negative dependence. Sample correlation coeﬃcients also have similar interpretation. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 56 / 122
57. 57. Multivariate data analysis basics Parameters and descriptive statistics Correlation matrix For a collection of p variables, the correlation matrix is a p × p matrix that displays the correlations between pairs of variables. For instance, the value in the j-th row and k-th column gives the correlation between variables xj and xk . The correlation matrix is symmetric so that the value in the k-th row and j-th column is also the correlation between variables xj and xk . The diagonal elements of the correlation matrix are all identically equal to 1. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 57 / 122
58. 58. Multivariate data analysis basics Parameters and descriptive statistics Correlation matrix The sample correlation matrix is denoted as R. R =      1 r12 . . . r1p r21 1 . . . r2p ... ... ... ... rp1 rp2 . . . 1      Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 58 / 122
59. 59. Multivariate data analysis basics Parameters and descriptive statistics Example The following covariance matrix show the pairwise covariances for the height, left forearm, left foot and head circumference measurements of n = 30 female college students. Here are two examples of calculating a correlation coeﬃcient: The correlation between height and left forearm is 3.022√ 8.74 √ 2.402 = 0.66. The correlation between head circumference and left foot is 0.118√ 3.434 √ 1.908 = 0.046. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 59 / 122
60. 60. Multivariate data analysis basics Parameters and descriptive statistics Example The complete sample correlation matrix for this example is the following: Overall, we see moderately strong linear associations among the variables height, left arm and left foot and quite weak (almost 0) associations between head circumference and the other three variables. In practice, use scatter plots of the variables to fully understand the associations between variables. It is not a good idea to rely on correlations without seeing the plots. Correlation values are aﬀected by outliers and curvilinearity. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 60 / 122
61. 61. Multivariate data analysis basics Parameters and descriptive statistics Overall measures of dispersion Sometimes it is also useful to have an overall measure of dispersion in the data. In this measure it would be good to include all of the variables simultaneously, rather than one at a time. In the past we looked at the individual variables and their variances to measure the individual variances. Here we are going to look at measures of dispersion of all variables together, particularly we are going to look at such measures that look at total variation. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 61 / 122
62. 62. Multivariate data analysis basics Parameters and descriptive statistics Overall measures of dispersion The variance σ2 j measures the dispersion of an individual variable Xj . The following two are used to measure the dispersion of all variables together. Total Variation Generalized Variance To understand total variation we ﬁrst must ﬁnd the trace of a square matrix. A square matrix is a matrix that has an equal number of columns and rows. Important examples of square matrices include the variance-covariance and correlation matrices. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 62 / 122
63. 63. Multivariate data analysis basics Parameters and descriptive statistics Overall measures of dispersion The trace of an n × n matrix A is trace(A) = n i=1 aii . For instance, in a 10 × 10 matrix, the trace is the sum of the diagonal elements. The total variation, therefore, of a random vector X is simply the trace of the population variance-covariance matrix. trace(Σ) = σ2 1 + σ2 2 + . . . σ2 p. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 63 / 122
64. 64. Multivariate data analysis basics Parameters and descriptive statistics Overall measures of dispersion Thus, the total variation is equal to the sum of the population variances. The total variation can be estimated by: trace(S) = s2 1 + s2 2 + · · · + s2 p . The total variation is of interest for principal components analysis and factor analysis and we will look at these concepts later in this course. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 64 / 122
65. 65. Multivariate data analysis basics Parameters and descriptive statistics Example Let us use the data from the USDA women’s health survey again to illustrate this. We have taken the variances for each of the variables from software output and have placed them in the table below. Variable Variance Calcium 157,829.4 Iron 35.8 Protein 934.9 Vitamin A 2,668,452.4 Vitamin C 5,416.3 Total 2,832,668.8 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 65 / 122
66. 66. Multivariate data analysis basics Parameters and descriptive statistics Example The total variation for the nutrient intake data is determined by simply adding up all of the variances for each of the individual variables. The total variation equals 2,832,668.8. This is a very large number. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 66 / 122
67. 67. Multivariate data analysis basics Parameters and descriptive statistics Example Problem: The problem with total variation is that it does not take into account correlations among the variables. These plots show simulated data for pairs of variables with diﬀerent levels of correlation. In each case, the variances for both variables are equal to 1, so that the total variation is 2. The corresponding variance-covariance matrix is: Σ = 1 ρjk σj σk ρjk σj σk 1 where ρjk is the correaltion coeﬃcient between the two variables. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 67 / 122
68. 68. Multivariate data analysis basics Parameters and descriptive statistics Example When the correlation r = 0, then we see a shotgun-blast pattern of points, widely dispersed over the entire range of the plot. Σ = 1 0 0 1 , being the Total Variation trace(Σ) = 2 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 68 / 122
69. 69. Multivariate data analysis basics Parameters and descriptive statistics Example Increasing the correlation to r = 0.7, we see an oval-shaped pattern. Note that the points are not as widely dispersed. Σ = 1 0.7 0.7 1 , being the Total Variation trace(Σ) = 2 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 69 / 122
70. 70. Multivariate data analysis basics Parameters and descriptive statistics Example Increasing the correlation to r = 0.9, we see that the points fall along a 45 degree line, and are even less dispersed. Σ = 1 0.9 0.9 1 , being the Total Variation trace(Σ) = 2 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 70 / 122
71. 71. Multivariate data analysis basics Parameters and descriptive statistics Generalized variance To take into account the correlations among pairs of variables an alternative measure of overall variance is suggested. This measure takes a large value when the various variables show very little correlation among themselves. In contrast, this measure takes a small value if the variables show very strong correlation among themselves, either positive or negative. This particular measure of dispersion is the generalized variance. In order to deﬁne the generalized variance we ﬁrst deﬁne the determinant of the matrix. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 71 / 122
72. 72. Multivariate data analysis basics Parameters and descriptive statistics Generalized variance We will start simple with a 2 × 2 matrix and then we will move on to more general deﬁnitions for larger matrices. Let us consider the determinant of a 2 × 2 matrix B as shown below. Here we can see that it is the product of the two diagonal elements minus the product of the oﬀ-diagonal elements. |B| = b11 b12 b21 b22 = b11b22 − b12b21. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 72 / 122
73. 73. Multivariate data analysis basics Parameters and descriptive statistics Generalized variance Here is an example of a simple matrix that has the elements 5, 1, 2 and 4. You will get the determinant 18. The product of the diagonal 5 × 4 subtracting the elements of the oﬀ-diagonal 1 × 2 yields an answer of 18: 5 1 2 4 = 5 × 4 − 1 × 2 = 20 − 2 = 18. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 73 / 122
74. 74. Multivariate data analysis basics Parameters and descriptive statistics Generalized variance More generally the determinant of a general p × p matrix B is given by the expression shown below: |B| = p j=1 (−1)j+1 b1j |B1j |. The expression involves the sum over all of the ﬁrst row of B. Note that these elements are noted by b1j . These are pre-multiplied by −1 raised to the (j + 1)-th power, so that basically we are going to have alternating plus and minus signs in our sum. The matrix B1j is obtained by deleting row i and column j from the matrix B. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 74 / 122
75. 75. Multivariate data analysis basics Parameters and descriptive statistics Generalized variance For a 3 × 3 matrix the determinant is: |B| = b11 b12 b13 b21 b22 b23 b31 b32 b33 = b11b22b33 + b12b23b31 + b13b21b32 −b13b22b31 − b32b23b11 − b33b21b12. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 75 / 122
76. 76. Multivariate data analysis basics Parameters and descriptive statistics Generalized variance By deﬁnition the generalized variance of a random vector X is equal to |Σ|, the determinant of the variance-covariance matrix. The generalized variance can be estimated by calculating |S|, the determinant of the sample variance-covariance matrix. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 76 / 122
77. 77. Multivariate data analysis basics Matrix manipulations for sample statistics Subsection 3 Matrix manipulations for sample statistics Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 77 / 122
78. 78. Multivariate data analysis basics Matrix manipulations for sample statistics The Mean and mean corrected data Suppose we have a n × p data matrix X and a 1 × n vector of ones, 1 . The mean vector is given by: ¯x = 1 n 1 X, and the mean corrected data are given by Xm = X − 1¯x where Xm gives the matrix containing the mean-corrected data. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 78 / 122
79. 79. Multivariate data analysis basics Matrix manipulations for sample statistics Sum of squares of cross products and covariance The matrix of sum of squares and cross products (SSCP) SSCPm is given by SSCPm = XmXm and the sample variance-covariance matrix is given by S = 1 n − 1 SSCPm = 1 n − 1 XmXm Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 79 / 122
80. 80. Multivariate data analysis basics Matrix manipulations for sample statistics Standardized data If we deﬁne a p × p diagonal matrix D, which has variances of the variables in the diagonal, then standardized data are given by Xs = XmD−1/2 . The SSCPs of standardized data is given by SSCPs = XsXs Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 80 / 122
81. 81. Multivariate data analysis basics Matrix manipulations for sample statistics Correlation matrix The correlation matrix is given by R = 1 n − 1 SSCPs = 1 n − 1 XsXs The generalized variance is given by |S| All these computations can be easily performed in R using the matrix operations... The following table presents and hypothetical data set Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 81 / 122
82. 82. Multivariate data analysis basics Matrix manipulations for sample statistics Firm Original Mean-corrected Standardized data data data X1 X2 X1 X2 X1 X2 1 13.000 4.000 7.917 3.833 1.619 1.108 2 10.000 6.000 4.917 5.833 1.006 1.686 3 10.000 2.000 4.917 1.833 1.006 0.530 4 8.000 -2.000 2.917 -2.167 0.597 -0.626 5 7.000 4.000 1.917 3.833 0.392 1.108 6 6.000 -3.000 0.917 -3.167 0.187 -0.915 7 5.000 0.000 -0.083 -0.167 -0.017 -0.048 8 4.000 2.000 -1.083 1.833 -0.222 0.530 9 2.000 -1.000 -3.083 -1.167 -0.631 -0.337 10 0.000 -5.000 -5.083 -5.167 -1.040 -1.493 11 -1.000 -1.000 -6.083 -1.167 -1.244 -0.337 12 -3.000 -4.000 -8.083 -4.167 -1.653 -1.204 Mean 5.083 0.167 0.000 0.000 0.000 0.000 SS 262.917 131.667 11.000 11.000 Var 23.902 11.970 23.902 11.970 1.000 1.000 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 82 / 122
83. 83. Multivariate data analysis basics Matrix manipulations for sample statistics Example The SSCP matrix of this data set is: SSCP = 262.917 136.375 136.375 131.667 The S matrix is: S = 1 n − 1 SSCP = 23.902 12.398 12.398 11.970 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 83 / 122
84. 84. Multivariate data analysis basics Matrix manipulations for sample statistics Example The correlation matrix R of these data is: R = s2 1 s1s1 s12 s1s2 s21 s2s1 s2 2 s2s2 = 1 0.733 0.733 1 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 84 / 122
85. 85. Multivariate data analysis basics A review of matrix deﬁnitions and operations Subsection 4 A review of matrix deﬁnitions and operations Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 85 / 122
86. 86. Multivariate data analysis basics A review of matrix deﬁnitions and operations Just a review! The following material reviews basic matrix deﬁnitions and operations. You should have previous knowledge on matrix algebra. This review does not relieve you from responsibility of being suﬃciently proﬁcient in these matters! Therefore be cautious and if you do not feel comfortable with these issues please get back again to these matters! Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 86 / 122
87. 87. Multivariate data analysis basics A review of matrix deﬁnitions and operations Transpose of a matrix The transpose of a matrix A is a matrix in which the rows of the transpose are the columns of A (so the columns of the transpose are the rows of A). The transpose of A is denoted as A . Example: for the 4 × 2 matrix, A =     3 6 2 5 3 7 1 8     the transpose is the 2 × 4 matrix A = 3 2 3 1 6 5 7 8 Notice that the columns of A are the rows of A Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 87 / 122
88. 88. Multivariate data analysis basics A review of matrix deﬁnitions and operations Symmetric matrices A square matrix A is symmetric if aij = aji , for all i and j. That is throughout the matrix, the element in the i-th row and j-th column equals the element in the j-th row and i-th column. Said another way, a matrix is symmetric when, A = A. Important examples of symmetric matrices in multivariate statistics include the variance-covariance matrix and the correlation matrix. These were deﬁned when we considered descriptive statistics. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 88 / 122
89. 89. Multivariate data analysis basics A review of matrix deﬁnitions and operations Symmetric matrices Example: the following is an example of a symmetric matrix. Notice that the ﬁrst row and the ﬁrst column are identical, the second row and the second column are the same, and that the third row is the same as the third column.   4 2 1 2 3 0 1 0 5   Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 89 / 122
90. 90. Multivariate data analysis basics A review of matrix deﬁnitions and operations Adding two matrices Two matrices may be added if and only if they have the same dimensions (same number of rows and also same number of columns as each other. To add two matrices, add corresponding elements (in terms of location). Example: 2 4 6 8 + 1 3 5 7 = 2 + 1 4 + 3 6 + 5 8 + 7 = 3 7 11 15 Note: one matrix is subtracted from another in the same way that matrices are added. To do the subtraction, subtract each element of the ﬁrst matrix from the corresponding (in location) element of the ﬁrst matrix Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 90 / 122
91. 91. Multivariate data analysis basics A review of matrix deﬁnitions and operations Multiplying a matrix by a scalar The word scalar is a synonym for a numerical constant. In matrix terms, a scalar is a matrix with one row and one column. To multiply a matrix by a scalar, multiply each element in the matrix by the scalar. Example: in this example, we multiply a matrix by the value 3. 3 × 2 4 6 8 = 3 × 2 3 × 4 3 × 6 3 × 8 = 6 12 18 24 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 91 / 122
92. 92. Multivariate data analysis basics A review of matrix deﬁnitions and operations Multiplication of matrices To be able to perform the matrix multiplication C = A × B, the number of columns in the matrix A must equal the number of rows in the matrix B. The element in the i-th row and j-th column of the answer matrix C, is the cross product sum of the i-th row of the matrix A and the j-th column of the matrix B. This is done of all combinations of rows of A and columns of B. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 92 / 122
93. 93. Multivariate data analysis basics A review of matrix deﬁnitions and operations Multiplication of matrices Example: we’ll carry out the multiplication A×B =   1 2 2 3 4 3  × 4 2 1 3 . The answer is C =   1 × 4 + 2 × 1 1 × 2 + 2 × 3 2 × 4 + 3 × 1 2 × 2 + 3 × 3 4 × 4 + 3 × 1 4 × 2 + 3 × 3   =   6 8 11 13 19 17   . Notice, for instance, that the element in the 1st row and 1st column of the answer C is the cross product sum of the 1st row of A and the 1st row of B. As another example, the 2nd row, 1st column element of C is the cross product sum of the 2nd row of A and the 1st column of B. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 93 / 122
94. 94. Multivariate data analysis basics A review of matrix deﬁnitions and operations The Identity Matrix An identity matrix is a square matrix that has the value one in each main diagonal position (from upper left to bottom right) and has the value 0 in all other locations. As an example of an identity matrix, the 3 × 3 identity matrix is I =   1 0 0 0 1 0 0 0 1   . I is called the identity matrix because multiplication of any square matrix A by the identity matrix yields the original matrix A as the answer. That is AI = IA = A. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 94 / 122
95. 95. Multivariate data analysis basics A review of matrix deﬁnitions and operations Matrix inverse An inverse (in the traditional sense) can be found only for square matrices. The inverse of a square matrix A is the matrix A−1 such that A−1 A = AA−1 = I. The calculation of an inverse for large matrices is a laborious process that we’ll leave to the computer. For 2 × 2 matrices, however, the formula is relatively simple. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 95 / 122
96. 96. Multivariate data analysis basics A review of matrix deﬁnitions and operations Matrix inverse For A = a11 a12 a21 a22 , the inverse is A−1 = 1 a11a22 − a12a21 a22 −a12 −a21 a11 . Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 96 / 122
97. 97. Multivariate data analysis basics A review of matrix deﬁnitions and operations Matrix inverse Example: we’ll determine the inverse of A = 10 6 8 5 . The inverse is A−1 = 1 10 × 5 − 6 × 8 5 −6 −8 10 = 1 2 5 −6 −8 10 = 2.5 −3 −4 5 . You might want to check that A−1 A = I. (It does!) Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 97 / 122
98. 98. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition Let A be a p × p square matrix and I be the p × p identity matrix. The scalars λ1, λ2, ..., λp satisfying the characteristic equation |A − λI| = 0 are called the eigenvalues (or characteristic roots) of matrix A. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 98 / 122
99. 99. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition On the left-hand side, we have the matrix A minus λ times the Identity matrix. When we calculate the determinant of the resulting matrix, we end up with a polynomial of order p. Setting this polynomial equal to zero, and solving for λ we obtain the desired eigenvalues. In general, we will have p solutions and so there are p eigenvalues, not necessarily all unique. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 99 / 122
100. 100. Multivariate data analysis basics A review of matrix deﬁnitions and operations Example Let A = 1 0 1 3 . Then |A − λI| = 1 0 1 3 − λ 1 0 0 1 = 1 − λ 0 1 3 − λ = (1 − λ)(3 − λ) implies that there are two roots, λ1 = 1 and λ2 = 3. The eigenvalues of A are 3 and 1 (the eigenvalues are in generally listed in descending order). Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 100 / 122
101. 101. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition The corresponding eigenvectors e1, e2, ..., ep are obtained by solving the expression below: (A − λj I)ej = 0 Here, we have the diﬀerence between the matrix A minus the j-th eigenvalue times the Identity matrix, this quantity is then multiplied by the j-th eigenvector and set it all equal to zero. This will obtain the eigenvector ej associated with eigenvalue λj . Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 101 / 122
102. 102. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition Note: This does not have a generally have a unique solution. So, to obtain a unique solution we will often require that ej transposed ej is equal to 1. Or, if you like, the sum of the square elements of ej is equal to 1, which means the length (norm) of the eigenvector is the unity. ej ej = 1 Also note that eigenvectors corresponding to diﬀerent eigenvalues are orthogonal. In situations, where two (or more) eigenvalues are equal, corresponding eigenvectors will still be orthogonal. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 102 / 122
103. 103. Multivariate data analysis basics A review of matrix deﬁnitions and operations Example (continued) The eigenvectors of matrix A associated to those eigenvalues (1 and 3) can be determined by solving the following equations: Ax = λ1x 1 0 1 3 x1 x2 = 1 × x1 x2 x1 x1 + 3x2 = x1 x2 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 103 / 122
104. 104. Multivariate data analysis basics A review of matrix deﬁnitions and operations Example (continued) There are many solutions! Therefore, setting x2 = 1 (arbitrarily) gives x1 = −2, and hence −2 1 is an eigenvector corresponding to the eigenvalue 1. After transforming for unit norm: e = (−2/ √ 5 1/ √ 5). Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 104 / 122
105. 105. Multivariate data analysis basics A review of matrix deﬁnitions and operations Example (continued) Ax = λ2x 1 0 1 3 x1 x2 = 3 × x1 x2 x1 x1 + 3x2 = 3x1 3x2 Again, setting x2 = 1 and x1 = 0 0 1 is an eigenvector corresponding to the eigenvalue 3. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 105 / 122
106. 106. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition Any symmetric square matrix can be reconstructed from its eigenvalues and eigenvectors. Let A be a p × p symmetric matrix. Then A can be expressed in terms of its p eigenvalue-eigenvector pairs (λi , ei ) as A = p j=1 λj ej ej Or, if you dispose the eigenvalues in a diagonal matrix Λ and the eigenvectors in a matrix E, then A = EΛE . Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 106 / 122
107. 107. Multivariate data analysis basics A review of matrix deﬁnitions and operations Example For example, let A = 2.2 0.4 0.4 2.8 Then |A − λI| = λ2 − 5λ + 6.16 − 0.16 = (λ − 3)(λ − 2) so A has eigenvalues λ1 = 3 and λ2 = 2. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 107 / 122
108. 108. Multivariate data analysis basics A review of matrix deﬁnitions and operations Example The corresponding eigenvectors are e1 = ( 1/ √ 5 2/ √ 5 ) and e2 = ( −2/ √ 5 1/ √ 5 ), respectively. Therefore A = 2.2 0.4 0.4 2.8 = 3 1/ √ 5 2/ √ 5 1/ √ 5 2/ √ 5 + +2 −2/ √ 5 1/ √ 5 −2/ √ 5 1/ √ 5 = 0.6 1.2 1.2 2.4 + 1.6 −0.8 −0.8 0.4 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 108 / 122
109. 109. Multivariate data analysis basics A review of matrix deﬁnitions and operations Example Or, A = 2.2 0.4 0.4 2.8 = = 1/ √ 5 −2/ √ 5 2/ √ 5 1/ √ 5 3 0 0 2 1/ √ 5 2/ √ 5 −2/ √ 5 1/ √ 5 = 3/ √ 5 −4/ √ 5 6/ √ 5 2/ √ 5 1/ √ 5 −2/ √ 5 2/ √ 5 1/ √ 5 = 3/5 + 8/5 6/5 − 4/5 6/5 − 4/5 12/5 + 2/5/ Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 109 / 122
110. 110. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition Usually the matrix A is taken to be either the variance-covariance matrix Σ, or the correlation matrix, or their estimates S and R, respectively. Eigenvalues and eigenvectors are used for: Computing prediction and conﬁdence ellipses Principal Components Analysis (later in the course) Factor Analysis (also later in this course) Discriminant analysis (also later in this course) Multidimensional scaling (also later in this course) ... Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 110 / 122
111. 111. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Consider the correlation matrix R as shown below: R = 1 ρ ρ 1 . Then, using the deﬁnition of the eigenvalues, we must calculate the determinant of R − λ times the Identity matrix. |R − λI| = 1 ρ ρ 1 − λ 1 0 0 1 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 111 / 122
112. 112. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Carrying out the math we end up with the matrix with 1 − λ on the diagonal and ρ on the oﬀ-diagonal. Then calculating this determinant we obtain 1 − λ squared minus ρ2 : 1 − λ ρ ρ 1 − λ = (1 − λ)2 − ρ2 = λ2 − 2λ + 1 − ρ2 . Setting this expression equal to zero we end up with the following... λ2 − 2λ + 1 − ρ2 = 0. Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 112 / 122
113. 113. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case To solve for λ we use the general result that any solution to the second order polynomial below: ay2 + by + c = 0 is given by the following expression: y = −b ± √ b2 − 4ac 2a . Here, a = 1, b = −2 (the term that precedes λ and c is equal to 1 − ρ2 . Substituting these terms in the equation above, we obtain that λ must be equal to 1 plus or minus the correlation ρ. λ = 2 ± 22 − 4(1 − ρ2) 2 = 1 ± 1 − (1 − ρ2) = 1 ± ρ Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 113 / 122
114. 114. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case We will take the following solutions: λ1 = 1 + ρ λ2 = 1 − ρ Next, to obtain the corresponding eigenvectors, we must solve a system of equations below: (R − λ1I)e1 = 0 and (R − λ2I)e2 = 0 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 114 / 122
115. 115. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Or in other words, this is translated for this speciﬁc problem in the expression below: 1 ρ ρ 1 − λ1 1 0 0 1 e11 e12 = 0 0 This simpliﬁes as follows: 1 − λ1 ρ ρ 1 − λ1 e11 e12 = 0 0 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 115 / 122
116. 116. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Yielding a system of two equations with two unknowns: (1 − λ1)e11 + ρe12 = 0 ρe11 + (1 − λ1)e12 = 0 Note that this does not have a unique solution. If (e11, e12) is one solution, then a second solution can be obtained by multiplying the ﬁrst solution by any non-zero constant c, i.e., (ce11, ce12). Therefore, we will require the additional condition that the sum of the squared values of e11 and e12 are equal to 1. e2 11 + e2 12 = 1 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 116 / 122
117. 117. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Consider the ﬁrst equation: (1 − λ1)e11 + ρe12 = 0 Solving this equation for e12 and we obtain the following: e12 = − (1 − λ1) ρ e11 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 117 / 122
118. 118. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Substituting this into e2 11 + e2 12 = 1 we get the following: e2 11 + (1 − λ1)2 ρ2 e2 11 = 1 Recall that λ = 1 ± ρ. In either case we end up ﬁnding that (1 − λ1)2 = ρ2 , so that the expression above simpliﬁes to: 2e2 11 = 1 Or, in other words: e11 = 1 √ 2 Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 118 / 122
119. 119. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Using the expression for e12 which we obtained above, e12 = − 1 − λ1 ρ e11 = 1 √ 2 we get e11 = 1 √ 2 and e12 = 1 √ 2 for λ1 = 1 + ρ Using the same reasoning we get e21 = 1 √ 2 and e22 = − 1 √ 2 for λ2 = 1 − ρ Therefore, the two eigenvectors are given by the two vectors as shown below: 1√ 2 1√ 2 for λ = 1 + ρ and 1√ 2 − 1√ 2 for λ = 1 − ρ Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 119 / 122
120. 120. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Some properties of the eigenvalues of the variance-covariance matrix are to be considered at this point. Suppose that λ1, λ2, ..., λp are the eigenvalues of the variance-covariance matrix Σ. By deﬁnition, the total variation is given by the sum of the variances. It turns out that this is also equal to the sum of the eigenvalues of the variance-covariance matrix. Thus, the total variation is: p j=1 s2 j = s2 1 + s2 2 + · · · + s2 p = λ1 + λ2 + · · · + λp = p j=1 λj Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 120 / 122
121. 121. Multivariate data analysis basics A review of matrix deﬁnitions and operations Eigenvalues, eigenvectors and spectral decomposition: a special case Some properties of the eigenvalues of the variance-covariance matrix are to be considered at this point. Suppose that λ1, λ2, ..., λp are the eigenvalues of the variance-covariance matrix Σ. The generalized variance is equal to the product of the eigenvalues: |Σ| = p j=1 λj = λ1 × λ2 × · · · × λp Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 121 / 122
122. 122. Multivariate data analysis basics A review of matrix deﬁnitions and operations Quadratic forms A quadratic form Q(x) in the p variables x1, x2, ...xp is Q(x) = x Ax, where x = [x1, x2, ..., xp] and A is a p × p symmetric matrix. Note that a quadratic form can be written as Q(x) = p i=1 p j=1 aij xi xj . For example, Q(x) = x1 x2 1 1 1 1 x1 x2 = x2 1 + x1x2 + x2 2 . Q(x) = x1 x2 x3   1 3 0 3 −1 −2 0 −2 2     x1 x2 x3   = x2 1 + 6x1x2 − x2 2 − 4x2x3 + 2x2 3 . Jorge M. Mendes (NOVAIMS) Multivariate Data Analysis Basics 2015 122 / 122