2. PCA Process – STEP 1: Get some data
• Subtract the mean from each of the
dimensions
• This produces a data set whose mean
is zero.
• Subtracting the mean makes variance
and covariance
• calculation easier by simplifying their
equations.
• The variance and co-variance values
are not affected by the mean value.
• Suppose we have two measurement
types X1 and X2, hence m = 2, and ten
samples each, hence n = 10.
4. PCA Process – STEP 2
• Calculate the covariance matrix
• Since the data is 2 dimensional, the covariance matrix will be 2x2.
• Since the non-diagonal elements in this covariance matrix are positive,
we should expect that both the X1 and X2 variables increase together.
• Since it is symmetric, we expect the eigenvectors to be orthogonal.
5. PCA Process – STEP 3
• Since the covariance matrix is square, we can calculate the eigenvectors and
eigenvalues for this matrix.
• Calculate the eigen vectors V and eigen values D of the covariance matrix
It is important to notice that these eigenvectors are both unit eigenvectors ie.
their lengths are both 1.
6. PCA Process – STEP 3
Eigenvectors are plotted as diagonal dotted
lines on the plot. (note: they are perpendicular
to each other).
One of the eigenvectors goes through the
middle of the points, like drawing a line of best
fit.
The second eigenvector gives us the other, less
important, pattern in the data, that all the
points follow the main line, but are off to the
side of the main line by some amount.
7. PCA Process – STEP 4
Reduce dimensionality and form feature
vector
• The eigenvector with the highest
eigenvalue is the principal component of
the data set.
• In our example, the eigenvector with the
largest eigenvalue is the one that points
down the middle of the data.
• Once eigenvectors are found from the
covariance matrix, the next step is to order
them by eigenvalue, highest to lowest. This
gives the components in order of
significance.
8. PCA Process – STEP 4
You need to form a feature vector, which is just a fancy name for a matrix of vectors. This is
constructed by taking the eigenvectors that you want to keep from the list of eigenvectors,
and forming a matrix with these eigenvectors in the columns.
9. STEP 4 - Deriving the new data set
• This the final step in PCA, and is also the easiest.
• Once we have chosen the components (eigenvectors) that we wish to keep
in our data and formed a feature vector, we simply take the transpose of
the vector and multiply it on the left of the original data set, transposed.
RowFeatureVector is the matrix with the eigenvectors in the columns
transposed so that the eigenvectors are now in the rows, with the most
significant eigenvector at the top.
RowZeroMeanData is the mean-adjusted data transposed, ie. the data items
are in each column, with each row holding a separate dimension.
Note: This is essential Rotating the coordinate axes so higher-variance axes come first.
10. • STEP 4 - Deriving the new data set
FinalData is the final data set, with data items in
columns, and dimensions along rows.
12. • So, how do we get the original data back?
• Before we do that, remember that only if we took all the eigenvectors in our
transformation will we get exactly the original data back.
• If we have reduced the number of eigenvectors in the final transformation, then the
retrieved data has lost some information.
• STEP 5 - Getting the old data back
13. • Recall that the final transform is this:
• which can be turned around so that, to get the original data back,
• STEP 5 - Getting the old data back
where is the inverse of . However, when we take all
the eigenvectors in our feature vector, it turns out that the inverse of our feature vector is
actually equal to the transpose of our feature vector. This is only true because the
elements of the matrix are all the unit eigenvectors of our data set.
But, to get the actual original data back, we need to add on the mean of that original data
(remember we subtracted it right at the start). So, for completeness,
14. This formula also applies to when you do not have all the eigenvectors in the
feature vector. So even when you leave out some eigenvectors, the above
equation still makes the correct transform.
• STEP 5 - Getting the old data back
