2) Entropy and Fisher Information

Question 1

What is the surprise / shannon information of an event

Answer 1

−log(𝑝)

Question 2

What is the log-odds ratio

Answer 2

Question 3

What is Entropy

Answer 3

A quantatitive measure of the distribution of probability mass. Low entropy means probability mass is concentrated

Question 4

What is Shannon Entropy of the Distribution P

Answer 4

Question 5

What are the Bounds for Shannon Entropy

Answer 5

Question 6

What is the difference between Shannon Entropy and Differential Entropy

Answer 6

• Shannon entropy is only defined for discrete random variables
• Differential Entropy results from applying the definition of Shannon entropy to a continuous random variable
• Differential entropy is not bounded below by zero and can be negative

Question 7

What is Differential Entropy

Answer 7

Question 8

What does the size of the entropy imply

Answer 8

Large entropy implies that the distribution is
spread out whereas small entropy means the distribution is concentrated

Question 9

What are maximum entropy distributions and what do they signify about a random variable

Answer 9

Maximum entropy distributions are considered minimally informative, indicating the distribution is highly spread out and thus less informative about the random variable’s outcomes
Key Examples Include -
* The discrete uniform distribution, representing maximum entropy among all discrete distributions.
* The normal distribution, as the maximum entropy distribution for continuous variables over [−∞, ∞] with specified mean and variance.
* The exponential distribution, for continuous distributions on [0, ∞] with a given mean.

Question 10

What is Cross-Entropy

Answer 10

Question 11

What is the Cross Entropy of Discrete and Continous Variables

Answer 11

Question 12

What is the key characteristic of cross-entropy between two distributions 𝐹 and 𝐺, and what does it simplify to when 𝐹 = 𝐺

Answer 12

• Cross-entropy is not symmetric with regard to 𝐹 and 𝐺, because the expectation is taken with reference to 𝐹
• If both distributions are identical cross-entropy reduces to Shannon and differential entropy, respectively: 𝐻(𝐹, 𝐹) = 𝐻(𝐹).

Question 13

What is Gibb’s Inequality

Answer 13

Question 14

What is KL Divergence / Relative Entropy

Answer 14

Question 15

What does KL Divergence measure

Answer 15

• 𝐷KL(𝐹, 𝐺) measures the amount of information lost if 𝐺 is used to approximate 𝐹.
• If 𝐹 and 𝐺 are identical (and no information is lost) then 𝐷KL(𝐹, 𝐺) = 0

Question 16

What are the properties of KL Divergence

Answer 16

• Asymmetry: 𝐷KL(𝐹, 𝐺) ≠ 𝐷KL(𝐺, 𝐹), i.e. the KL divergence is not symmetric, 𝐹 and 𝐺 cannot be interchanged.
• Zero Equivalence: 𝐷KL(𝐹, 𝐺) = 0 if and only if 𝐹 = 𝐺
• Non-negativity: 𝐷KL(𝐹, 𝐺) ≥ 0
• Coordinate Invariance: 𝐷KL(𝐹, 𝐺) remains invariant under coordinate transformations

Question 17

How can the KL divergence be locally approximated using Expected Fisher Information

Answer 17

Question 18

Define Expected Fisher Information

Answer 18

Question 19

What is the general form of the Expected Fisher Information matrix for models with multiple parameters, including its gradient

Answer 19