Lecture 17

Question 1

PCA what is it? (+/-)

Answer 1

reduce dimension, combine the feature variables in a specific way, retaining the most valuable parts of all of the feature variables, each of the “new variables” after PCA are all independent of one another (if strong correlation:reduction of dimension)
cons: loss of interpretability

Question 2

PCA how to overall

Answer 2

Find the directions of maximum variance in high- dimensional dataset (n dimension) and project it onto a subspace with smaller dimension (k dimension, with k < n), while retaining most of the information.

Question 3

PCA eq

Answer 3

A = X.T@X (nxn) - covariance matrix
Diagonalize A=UDU^{-1} (D:λ1, …, λn ; U:u1, …, un)
New feature X* = XU such that X.T@X=D
X* first 2 columns (principal components) explain most of the variance (λ1/λtotal + λ2/λtotal)

Question 4

Diagonalization

Answer 4

A nxn with n linearly independent eigenvector u, then A=UDU^{-1} (U lin indep normalised eigenvectors, D eigenvalues)
If less than n lin indep eigenvector, defective and not diagonalizable

Question 5

nxn symmetric matrix A with n distinct eigenvalues ?

Answer 5

Hence diagonalizable (/!\ if A diagonalizable, not necessarily distinct eigenvalues)

Question 6

Identity matrix diagonalization

Answer 6

Iv = λv
I = UDU-1 with eigenvalues all 1 and u1=(1 0 ... 0), u2=(0 1 0 ... 0), ...

Question 7

SVD

Answer 7

Factorization mxn matrix A=UΣV.T where U mxm orthogonal, V.T nxn orthogonal, Σ mxn diagonal (σ1 >= σ2 >= …)

Question 8

SVD, V?

Answer 8

A.T@A = V@Σ^2@V.T

V eigenvectors of A.T@A (lin. indep - vi.vj=0, right singular vectors)

Question 9

SVD, A@A.T eigenvectors?

Answer 9

A@A.T = U@Σ^2@U.T

Columns of U are eigenvectors of A@A.T (left singular vectors)

Question 10

SVD computation

Answer 10

1) A.T@A
2) Get V, D = eigen(A.T@A) (V columns right singular vector, D eigenvalues)
3) σi = √λi or Σ=D^{1/2} (singular values)
4) U = AVΣ^{-1} (columns are left singular vectors)

Question 11

A.T@A eigenvalues

Answer 11

Non-negative because semi positive definite and symmetric (always square root!)

Question 12

Orthogonal singularity ?

Answer 12

Not singular (X^{-1} = X.T)

Question 13

Σ can have zero diagonal

Answer 13

Yes, singular value can be zero

Question 14

Singular value of A ?

Answer 14

Squared singular values of A are eigenvalues of A.T@A (σi = √λi)