6) Optimality properties and conclusion

Question 1

What is a summary statistic T(D) and how can it vary in its representation?

Answer 1

A function of the observed data {D} = {x 1,…,x n }
designed to describe key characteristics of the data. It can take various forms, including scalar values, vectors, matrices ect
Summary statistics typically focus on important data properties like:
* Location: For example, the mean (ˉx ) or the median
* Scale: Such as the standard deviation or the interquartile range.

Question 2

When is a statistic sufficient

Answer 2

• If the corresponding likelihood function can be written using only 𝑇(𝐷) in the terms that involve 𝜽 such that
• 𝐿(𝜽|𝐷) = ℎ(𝑇(𝐷), 𝜽) 𝑘(𝐷) ,
  where ℎ() and 𝑘() are positive-valued functions, and or equivalently on log-scale
• 𝑙𝑛(𝜽) = log ℎ(𝑇(𝐷), 𝜽) + log 𝑘(𝐷)

Question 3

How does the existence and uniqueness of the MLE
relate to sufficient statistics

Answer 3

If the MLE exists and is unique, then θ^ ML is a unique function of the sufficient statistic

Question 4

How does a sufficient statistic partition the space of data sets

Answer 4

A sufficient statistic effectively partitions the space of all possible data sets into clusters, where each cluster contains data sets that result in the same value of
T(D). This partitioning is represented by:

The data sets in 𝒳𝑡 are equivalent in terms of the sufficient statistic

Question 5

What does it mean for two data sets to be likelihood equivalent

Answer 5

Two data sets 𝐷1 and 𝐷2 for which the ratio of the corresponding likelihoods 𝐿(𝜽|𝐷1)/𝐿(𝜽|𝐷2) does not depend on 𝜽

Question 6

Is 𝒳𝑡 liklihood equivalent

Answer 6

Yes all data sets in 𝒳𝑡 are likelihood equivalent

Question 7

What defines a minimal sufficient statistic

Answer 7

A minimal sufficient statistic is defined as a sufficient statistic for which all likelihood equivalent data sets are also equivalent under this statistic

Question 8

What is a trivial example of a minimal sufficient statistic

Answer 8

The likelihood function itself since by definition it can be computed from any set of sufficient statistics

Question 9

What are the differences between forward KL and reverse KL divergence minimisation

Answer 9

• Forward KL Divergence, min𝜽 𝐷KL(𝐹0, 𝐹𝜽)
  zero avoiding property: 𝑓𝜽(𝑥) > 0 whenever 𝑓0(𝑥) > 0
• Reverse KL Divergence, min𝜽 𝐷KL(𝐹𝜽, 𝐹0)
  zero forcing property: 𝑓𝜽(𝑥) = 0 whenever 𝑓0(𝑥) = 0

Question 10

How does a small sample size n affect the reliability of MLE and what are alternative strategies

Answer 10

MLE can overfit, meaning it too closely matches the particularities of the small dataset, leading to poor generalization to the broader population.
Alternative methods -
* Regularised/penalised likelihood
* Bayesian methods