Week 10 Variational inference

Question 1

Formulate ELBO

Answer 1

Question 2

How to restrict proposal

Answer 2

Question 3

How to optimize q(Z)

Answer 3

Where we find q(Z) through factorising (or parametric?)

Question 4

How to construct quantity to be optimized for ELBO

Answer 4

Where 10.3 is L(q) = integral( q(Z) * ln{ (p(X,Z)) / (q(Z)} dZ )

And 10.5 is q(Z) = Πq_i(z_i) for i =1,.., M (factorizing)

Question 5

Construct general expression for optimal solution to ELBO

Answer 5

Crucially the constant comes from normalizing qi(Zi)

Question 6

General factorized approximations (Gaussian example)

Answer 6

VI: factorized approximation of posterior
Gfa: approximating general distribution by a factorized distribution

Where p(z) over z = (z1, z2), which are correlated, has μ = (μ1, μ2) and Λ = ((Λ, Λ), (Λ, Λ))
Then we produce the optimal proposals by using the general form of ln(qj*(Zj))

Question 7

Constructing form of optimal proposals for factorized approximation of bivariate Gaussian

Answer 7

Question 8

Reverse KL

Answer 8

Gaussian example is general factorisation approximation of bivariate gaussian

Question 9

Alpha family

Answer 9

The 2 forms of KL divergence are members of the Alpha family of

Question 10

Setup VI for univariate Gaussian

Answer 10

Question 11

Optimisation for VI for univariate Gaussian

Answer 11

Question 12

Iterative solution for VI for univariate Gaussian

Answer 12

Question 13

Decompose variational distribution for model comparison

Answer 13

Question 14

Lower bound on ln(p(x)) in model comparison

Answer 14

Where we assume discrete Z but the same analysis applies to continuous latent variables provided summations are replaced with integrations

Question 15

Calaculating optimal L_m for model comparison

Answer 15