3) Maximum Likelihood Estimation

Question 1

What is the Likelihood Function and Log-Likelihood Function

Answer 1

Question 2

What is the score function

Answer 2

The first derivative of the log-likelihood function with regard to the parameters

Question 3

What steps are involved in finding the Maximum Likelihood Estimate (MLE)

Answer 3

• The condition for MLE is 𝑺𝑛(𝜽ˆ 𝑀𝐿) = 0 indicating the point where the log-likelihood function’s slope is zero, suggesting a maximum or minimum.
• Confirming a Maximum:
• Scalar θ: Ensure the second derivative of the log-likelihood is negative confirming maximum at 𝜽ˆ𝑀𝐿
• Vector θ: Compute the Hessian matrix at the MLE. The MLE is confirmed if this matrix is negative definite (all eigenvalues are negative), ensuring the log-likelihood is locally concave.

Question 4

How can cross-entropy and KL divergence be approximated for large sample sizes

Answer 4

For large sample size n, the cross-entropy 𝐻(𝐹, 𝐺𝜽) is approximated by the empirical cross-entropy where the expectation is taken with regard to 𝐹ˆ𝑛 rather than 𝐹

Question 5

How is maximizing the log-likelihood related to minimizing KL divergence for large sample sizes

Answer 5

Question 6

What does it mean for an estimator to be consistent

Answer 6

Question 7

Does the maximum likelihood function produce consistent estimates

Answer 7

Yes the method of maximum likelihood in general produces consistent estimates

Question 8

Why are MLEs considered asymptotically unbiased

Answer 8

Maximum Likelihood Estimates (MLEs) are considered asymptotically unbiased because, with sufficient data, the MLE will converge to the true parameter value

Question 9

What is the relationship of maximum likelihood with least squares estimation

Answer 9

Finding the mean by maximum likelihood assuming a normal model is equivalent to least-squares estimation

Question 10

What is the observed Fisher information

Answer 10

The negative curvature at the MLE:
𝑱𝑛(𝜽ˆ𝑀𝐿), the derivative of the negative score function evaluated at the MLE

Question 11

What is the relationship between observed and expected Fisher information

Answer 11