Mathematical Concept - Standard Deviation - Covariance - Eigenvector - Eigenvalues

INTRODUCTION
 PCA is a useful statistical techniques that has found application in

field that has found application in field such as a face recognition
and image compression, and is a common technique for finding
pattern in data of high dimension.
• Mathematical Concept
• Standard deviation
• Covariance
• Eigenvector
• Eigenvalues.
A standard deviation (or σ) is a measure of how dispersed the data is in
relation to the mean. Low standard deviation means data are clustered
around the mean, and high standard deviation indicates data are more
spread out.
The standard deviation is the average amount of variability in your data

set. It tells you, on average, how far each score lies from the mean.
• Step 1: Find the mean.
• Step 2: For each data point, find the square of its distance to the
mean.
• Step 3: Sum the values from Step 2.
• Step 4: Divide by the number of data points.

• The standard deviation measures the spread of the data about the
mean value. It is useful in comparing sets of data which may have
the same mean but a different range.
• For example, the mean of the following two is the same:
• 15, 15, 15, 14, 16 and 2, 7, 14, 22, 30.
• However, the second is clearly more spread out.

• The variance is a measure of variability.
• It is calculated by taking the average of squared deviations from the mean.
• Variance tells you the degree of spread in your data set.
• The more spread the data, the larger the variance is in relation to the
mean.
• How to calculate
• Finding the mean(the average).
• Subtracting the mean from each number in the data set and then squaring
the result. The results are squared to make the negatives positive. ...
• Averaging the squared differences.
Covariance
• Covariance is a statistical tool that is used to determine the
relationship between the movements of two random variables.
When two stocks tend to move together, they are seen as having a
positive covariance; when they move inversely, the covariance is
negative.
• Covariance indicates the relationship of two variables whenever one
variable changes. If an increase in one variable results in an increase
in the other variable, both variables are said to have a positive
covariance. Decreases in one variable also cause a decrease in the
other.
• An eigenvector is a vector that when multiplied with a
transformation matrix results in another vector multiplied with a
scaler multiple having the same direction as Eigenvector. This scaler
multiple is known as Eigenvalue.
• The eigenvalue is a measure of how much of the common variance

of the observed variables a factor explains.
Algorithm steps
Step 1: Get your data

Separate your data set into Y and X. Y will be the validation set and X will be the training set. In
simple terms, we will use X for our study and use Y to check whether our study is correct.
Step 2: Give your data a structure
Take the 2 dimensional matrix of independent variables X. Rows represent data items and
columns represent features. The number of columns is the number of dimensions.
For each column, subtract the mean of that column from each entry. (This ensures that each
column has a mean of zero.)
Step 3: Standardize your data
Given the columns of X, are features with higher variance more important than features with lower
variance, or is the importance of features independent of the variance? (In this case, importance
means how well that feature predicts Y.)
If the importance of features is independent of the variance of the features, then divide each
observation in a column by that column’s standard deviation. Call the centered and standardized
matrix Z.
Step 4: Get Covariance of Z
Take the matrix Z, transpose it and multiply the transposed matrix by Z.
Convariance of Z = ZᵀZ
The resulting matrix is the covariance matrix of Z, up to a constant.
Step 5: Calculate Eigen Vectors and Eigen Values
Calculate the eigenvectors and their corresponding eigenvalues of ZᵀZ.
The eigendecomposition of ZᵀZ is where we decompose ZᵀZ into PDP⁻¹,

where P is the matrix of eigenvectors
D is the diagonal matrix with eigenvalues on the diagonal and values of zero everywhere else.
The eigenvalues on the diagonal of D will be associated with the corresponding column in P — that is,
the first element of D is λ₁ and the corresponding eigenvector is the first column of P. This holds for all
elements in D and their corresponding eigenvectors in P. We will always be able to calculate PDP⁻¹ in
this fashion.
Step 6: Sort the Eigen Vectors
λ₁, λ₂, …, λp and sort them from largest to smallest. In doing so, sort the eigenvectors in P
Take the eigenvalues
accordingly. (For example, if λ3 is the largest eigenvalue, then take the third column of P and place it
in the first column position.)
Call this sorted matrix of eigenvectors P*. The columns of P* are the same as the columns of P in a
different order. Note that these eigenvectors are independent of one another.
Step 7: Calculate the new features
Calculate Z* = ZP*.
This new matrix, Z*, is a centered/standardized version of X but now each observation is a
combination of the original variables, where the weights are determined by the eigenvector. As a
bonus, because our eigenvectors in P* are independent of one another, each column of Z* is also
independent of one another.
Step 8: Drop unimportant features from the new set

Mathematical Concept - Standard Deviation - Covariance - Eigenvector - Eigenvalues

Uploaded by

Copyright:

Available Formats

Mathematical Concept - Standard Deviation - Covariance - Eigenvector - Eigenvalues

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mathematical Concept - Standard Deviation - Covariance - Eigenvector - Eigenvalues

Uploaded by

Copyright:

Available Formats

INTRODUCTION

 PCA is a useful statistical techniques that has found application in

The standard deviation is the average amount of variability in your data

• Step 4: Divide by the number of data points.

• However, the second is clearly more spread out.

• The eigenvalue is a measure of how much of the common variance

Step 1: Get your data

The eigendecomposition of ZᵀZ is where we decompose ZᵀZ into PDP⁻¹,

You might also like