Bayesian inference

Question 1

Define prior and posterior distribution, and give formulas for the posterior distribution of π(θ|x)

Answer 1

The prior distribution π(θ) of θ is the probability distribution of θ before observing the data. It represents our beliefs or uncertainty about the parameter before collecting any data. After observing data X = x, we update the distribution of θ to obtain the posterior distribution π(θ|x) representing our updated beliefs in light of seeing x.
pg 31

Question 2

What is an improper prior?

Answer 2

A non-negative prior function where the integral over the parameter space is not finite.

Question 3

Define a Jeffreys prior. Is it always proper?

Answer 3

Prior proportional to sqrt[ det( I(θ) ) ]
No
pg 34

Question 4

Define a loss function

Answer 4

Non negative function that determines the cost of a particular action for a given parameter θ

Question 5

Define the risk function for loss function L and decision rule δ

Answer 5

E[ L(δ(X), θ) ] = integral

pg 37

Question 6

When is a decision rule δ inadmissible?

Answer 6

pg 38

Question 7

Define the π-Bayes risk for decision rule δ

Answer 7

pg 38

The estimator that minimizes this risk is called the Bayes estimator

Question 8

What is the posterior risk?

Answer 8

The average loss under the posterior distribution for an observation X
pg 39

Question 9

Does minimizing the posterior risk also minimize the π-Bayes risk?

Answer 9

Yes (proof on pg 39)

Question 10

What is the minimax risk?

Answer 10

The minimax risk is defined as the infimum (‘min’) over all decision rules δ of the maximal (‘max’) risk over the whole parameter space Θ
pg 40

Question 11

What happens if a Bayes rule δ has constant risk in θ?

Answer 11

If a (unique) Bayes rule δ has constant risk in θ then it is (unique) minimax.

Question 12

What is a uniformly minimum variance unbiased estimator?

Answer 12

An unbiased estimator g^(X) of g(θ) s.t. var(g^) <= any other unbiased estimator g(X) of g(θ)

Question 13

What does it mean to say a statistic is complete for θ?

Answer 13

if E_θ [g(T)] = 0 for all θ then P_θ (g(T)=0) = 1

pg43

Question 14

Give an example of a complete statistic for a k-parameter exponential family

Answer 14

T = ( T1(X), …, Tk(X) )

Question 15

Can a sufficient, complete satistic be minimal?

Answer 15

If a sufficient statistic T is complete, then it is minimal, but not all minimal sufficient statistics are complete

Question 16

Define ancillary statistic

Answer 16

A statistic is an ancillary statistic if its distribution does not depend on the parameter θ.

Question 17

State Basu’s theorem

Answer 17

If T is a complete sufficient statistic for θ, then any ancillary statistic V is independent of T.

Question 18

State the Lehmann-Scheffe theorem.

Answer 18

Let T be a sufficient and complete statistic for θ and
g˜ be an unbiased estimator of g(θ) with var_θ(˜g) < ∞ for all θ∈ Θ. If gˆ(T(X)) = E[˜g(X)|T(X)],
then gˆ is the unique uniformly minimum variance unbiased estimator (UMVUE) of g(θ).

Question 19

What is the likelihood principle?

Answer 19

The likelihood principle says that all bayesian inferences are based on the likelihood function only.
ie. if the likelihood function with two different thetas are the same then all their inferences (prior, MLE,…) are the same

Question 20

How can we prove T is not complete?

Answer 20

Find a function g such that E(g(T))=0 but g(T) =/= 0