week 7

Question 1

“What is the core idea behind Data Augmentation (DA) algorithms?”

Answer 1

To simplify sampling from a target distribution π(x) by introducing auxiliary variables y and sampling from an augmented joint density f(x, y) using Gibbs sampling.

Question 2

“In the formal setting of DA, what is the relationship between the target distribution π(x) and the augmented density f(x, y)?”

Answer 2

The target distribution is the marginal of the augmented density: ∫ f(x, y) dy = π(x).

Question 3

“What is the augmented target distribution?”

Answer 3

The joint density f(x, y) constructed such that its marginal distribution for x is the original target distribution π(x).

Question 4

“What is the general structure of a DA algorithm iteration, starting with current state x’ = x(i)?”

Answer 4

Step 1: Sample auxiliary variable y ~ fY|X(· | x’). Step 2: Sample target variable x ~ fX|Y(· | y). Step 3: Set the next state x(i+1) = x.

Question 5

“What happens to the auxiliary variable y sampled in Step 1 of a DA iteration?”

Answer 5

It is discarded after Step 2 (or 3), as it is not part of the original target distribution.

Question 6

“When is a Metropolis-within-Gibbs step needed within a DA algorithm?”

Answer 6

When the full conditional distributions fX|Y or fY|X are not known distributions from which direct sampling is easy.

Question 7

“What is the key idea of the Slice Sampler, a type of DA algorithm?”

Answer 7

Sampling from π(x) is equivalent to sampling uniformly from the region S under the graph of π(x), specifically S = {(x, u) : 0 ≤ u ≤ π(x)}.

Question 8

“What is the augmented target density f(x, u) used in the Slice Sampler?”

Answer 8

f(x, u) is proportional to the indicator function 1S(x, u), meaning it’s uniform over the set S = {(x, u) : 0 ≤ u ≤ π(x)}.

Question 9

“Describe Step 1 of the Slice Sampler algorithm, given the current state x’ = x(i).”

Answer 9

Simulate the auxiliary height variable u ~ Uniform(0, π(x’)).

Question 10

“Describe Step 2 of the Slice Sampler algorithm, given the sampled height u.”

Answer 10

Simulate the next state x ~ Uniform(A), where A is the ‘slice’ A = {z : π(z) > u}.

Question 11

“What is the probability mass function f(k; ρ) for the Yule-Simon distribution?”

Answer 11

f(k; ρ) = ρ * B(k, ρ + 1), for k = 1, 2, … and shape parameter ρ > 0, where B(·, ·) is the Beta function.

Question 12

“How can the Yule-Simon distribution be represented as a mixture?”

Answer 12

If W ~ Exponential(ρ) and K | W ~ Geometric(e⁻ʷ), then the marginal distribution of K is Yule-Simon(ρ).

Question 13

“What is the integral representation of the Yule-Simon PMF derived from its mixture representation?”

Answer 13

f(k; ρ) = ∫[0 to ∞] e⁻ʷ * (1 - e⁻ʷ)^(k-1) * ρ * e⁻ρʷ dw

Question 14

“In the DA approach for the Yule-Simon distribution, what is the augmented target distribution f(k, w)?”

Answer 14

f(k, w) = e⁻ʷ * (1 - e⁻ʷ)^(k-1) * ρ * e⁻ρʷ, for w > 0 and k = 1, 2, ….

Question 15

“In the DA approach for Yule-Simon, what is the full conditional distribution of K given W=w?”

Answer 15

K | W=w ~ Geometric(e⁻ʷ).

Question 16

“In the DA approach for Yule-Simon, what is the full conditional distribution f(w|k) proportional to?”

Answer 16

f(w|k) ∝ e⁻ʷ * (1 - e⁻ʷ)^(k-1) * e⁻ρʷ.

Question 17

“By using the change of variable T = e⁻ʷ, what is the full conditional distribution T | K in the Yule-Simon DA?”

Answer 17

T | K ~ Beta(ρ + 1, K).

Question 18

“Describe Step 1 of the DA algorithm for sampling from Yule-Simon(ρ), given current state k(i).”

Answer 18

Sample t(i+1) | k(i) ~ Beta(ρ + 1, k(i)).

Question 19

“Describe Step 2 of the DA algorithm for sampling from Yule-Simon(ρ), given t(i+1).”

Answer 19

Compute w(i+1) = -log(t(i+1)).

Question 20

“Describe Step 3 of the DA algorithm for sampling from Yule-Simon(ρ), given w(i+1).”

Answer 20

Sample k(i+1) ~ Geometric(e⁻ʷ⁽ⁱ⁺¹⁾).

Question 21

“What defines the posterior distribution in Bayesian statistics?”

Answer 21

π(θ|y) ∝ π(θ)L(θ; y), where π(θ) is the prior and L(θ; y) is the likelihood.

Question 22

“Under what condition might the likelihood function L(θ; y) be intractable?”

Answer 22

When it involves integrals over latent variables (e.g., mixed models) or complex normalizing constants (e.g., Potts model, models on graphs).

Question 23

“In the general mixed-effects model yjk = xᵀjkβ + bk + εjk, why is the likelihood f(yn|θ) often intractable?”

Answer 23

It requires integrating out the random effects bk over their distribution Fb.

Question 24

“In the Potts model, why is the likelihood f(yn|θ) intractable?”

Answer 24

The normalizing constant involves a sum over all kⁿ possible configurations of the n pixels, which is computationally infeasible for large n.

Question 25

"What is the likelihood (PMF) for the Wallenius distribution P(x; n, m, ω)?"

Answer 25

P(x; n, m, ω) = [Π(j=1 to c) (mj choose xj)] * ∫[0 to 1] Π(j=1 to c) [1 - t^(ωj/d)]^xj dt, where d = Σωj(mj-xj).

Question 26

"Why is sampling from the Wallenius distribution (or its posterior) not straightforward?"

Answer 26

The likelihood involves an integral that is generally not available in closed form.

Question 27

"What is the key requirement for applying Approximate Bayesian Computation (ABC) methods?"

Answer 27

The ability to simulate data z from the likelihood model f(·|θ) for any given parameter θ.

Question 28

"What is the core step in the exact likelihood-free rejection sampling algorithm (for discrete data y)?"

Answer 28

Accept a parameter θ' drawn from the prior π(θ) if data z simulated from f(·|θ') exactly matches the observed data y (z=y).

Question 29

"Why does exact likelihood-free rejection sampling typically fail for continuous data?"

Answer 29

The probability of simulating z that exactly matches y (P(z=y | θ')) is typically zero.

Question 30

"How does ABC modify the acceptance condition of likelihood-free rejection sampling?"

Answer 30

It accepts θ' if the simulated data z is 'sufficiently close' to the observed data y, measured by a distance d(·) being less than a tolerance ε.

Question 31

"How are summary statistics s(·) used in ABC, especially for high-dimensional data?"

Answer 31

Instead of comparing full datasets z and y, ABC compares their summary statistics: d(s(z), s(y)) ≤ ε.

Question 32

"What is the acceptance condition in the standard ABC rejection sampling algorithm using summary statistics?"

Answer 32

d{s(z), s(y)} ≤ ε, where z ~ f(·|θ'), θ' ~ π(θ), s(·) is the summary statistic function, d(·,·) is the distance function, and ε is the tolerance.

Question 33

"The accepted values θ from ABC rejection sampling are i.i.d. draws from which distribution?"

Answer 33

They are draws from the approximate posterior distribution π_ε(θ|y), which approximates the true posterior π(θ|y).

Question 34

"What distribution do the accepted pairs (θ, z) follow in ABC rejection sampling?"

Answer 34

π_ε(θ, z|y) ∝ π(θ)f(z|θ)IAε,y(z), where IAε,y(z) is the indicator function for the acceptance region {z : d(s(z), s(y)) ≤ ε}.

Question 35

"What are the three critical choices that need to be made when implementing ABC?"

Answer 35

1. Choice of summary statistics s(y), 2. Choice of distance function d(·,·), 3. Choice of tolerance level ε.

Question 36

"In the ABC example for the Wallenius distribution, what summary statistic was used?"

Answer 36

The average frequency vector across k draws: p̂ = (1/k) Σ[h=1 to k] p̂h, where p̂h is the vector of observed frequencies (xhj/nh) for draw h.

Question 37

"In the ABC example for the Wallenius distribution, what distance metric was used?"

Answer 37

Distance in variation: ρ(p̂(l), p̂(t)) = (1/2) Σ[j=1 to c] | p̂j(l) - p̂j(t) |.

Question 38

"What prior was used for the parameter ω in the Wallenius ABC example?"

Answer 38

A symmetric Dirichlet prior.