Week 2 (MLE, MAP, Bayesian Curve, Entropy)

Question 1

Stirling’s Approximation

Answer 1

As N -> inf

Question 2

Marginal probability, Joint probability and conditional probability in the context of the grid

Answer 2

Where N is the total number of possibilities

Question 3

Sum rule and product rule in context of grid

Answer 3

Question 4

Transformed densities formula

Answer 4

Question 5

Variance and covariance formulae

Answer 5

Question 6

Multivariate Gaussian PDF

Answer 6

Question 7

Log Likelihood and estimators of μ and σ^2 for univariate Gaussian

Answer 7

Question 8

Expectation of MLEs for univariate Gaussian

Answer 8

Question 9

Connect the MLE to polynomial curve fitting and construct resulting likelihood function

Answer 9

Instead of taking the naive error minimisation approach we let t_n = y(x_n, w) + ε_n where ε is Gaussian nose distributed N(0, β^-1)

This implies p(t_n | x_n, w, β) = Π_n N(y(x_n, w), β ^-1)

Question 10

Apply MLE to polynomial curve fitting to produce predictive distribution

Answer 10

Question 11

How does MAP differ from MLE predictive distribution

Answer 11

We introduce a prior on the weights w addressing overfitting by regularisation

Question 12

Process for MAP

Answer 12

Where p(w| α) is the Gaussian prior we assume on w

Question 13

How does Bayesian curve fitting build on MAP

Answer 13

We treat w as a RV computing the full posterior and integrating over all possible w to make predictions

This addresses the limitations of the MLE and MAP approaches by quantifying the uncertainty of both the data and the model

Question 14

Process for Bayesian curve fitting

Answer 14

Where p(w| x, t, α, β) is the posterior (as it is for MAP)

Question 15

Entropy equation discrete

Answer 15

Question 16

Cross validation, method & purpose

Answer 16

To select best degree of polynomial curve fitting split data into training and validation sets multiple times for each model for i=1,…, M :
Train model on training set
Evaluate error on validation sets
Choose M with lowest error

To avoid overfitting by over parametrising

Question 17

Differential entropy for continuous variables

Answer 17

Question 18

Differential entropy maximised for Gaussian distribution

Answer 18

Question 19

Conditional entropy

Answer 19

Question 20

Def KL divergence

Answer 20

Question 21

Def mutual information

Answer 21

Mutual information I[x,y] measures information shared between x and y

Question 22

Decision theory process

Answer 22

Question 23

Minimum expected loss for classification

Answer 23

Question 24

Decision theory steps for regression

Answer 24

Question 25

Derivation of squared loss function

Answer 25

Question 26

Generative vs Discriminative approaches

Answer 26

Generative models learn the full join distribution allowing density estimation and handling missing data Discriminative models focus on the decision boundary, often simpler but less flexible

Question 27

Notation of bayesian curve fitting

Answer 27