Week 5 (GMM)

Question 1

Combining simplex models to form GMM

Answer 1

Question 2

Log likelihood for GMM

Answer 2

Question 3

Latent variable representation for GMM

Answer 3

Question 4

Form responsibility for GMM

Answer 4

Question 5

Derive MLE for μ for GMM

Answer 5

Question 6

Derive MLE for Σ for GMM

Answer 6

Question 7

MLE for mixing term for GMM

Answer 7

Question 8

Put together all MLE for GMM

Answer 8

Question 9

EM algorithm for GMM

Answer 9

Repeat:
1) calculate responsibility for each pairwise combo of n and k
2) update MLEs using prev step
3) update param values using MLEs

Question 10

Problems with GMM

Answer 10

Singularity - a mixture component collapses on a data point

Identifiability - MLE solution in a K component mixture has K! Solutions due to permutation symmetry

Question 11

Crucial difference between EM and Variational inference

Answer 11

The parameters become stochastic variables in VI and therefore their parameter vector θ doesn’t appear in the ELBO decomposition (whereas it does in that fro EM)