Entropy

Question 1

What is entropy?

Answer 1

Layman: measure of disorder in a closed system

Information theory: a measure of the lack of structure or detail in the probability distribution describing your knowledge

Question 2

What is maximum entropy in practice?

Answer 2

Finding the flattest possible probability distribution that is compatible with the constraints that arise from prior knowledge.

Question 3

If given a distribution with lower entropy versus one with higher, what is the justification for choosing the one with higher entropy?

Answer 3

If both of these distributions are consistent your prior knowledge, you should prefer the one with higher information entropy because it makes fewer implicit unwarranted assumptions.

Systems also tend to develop towards maximum entropy.

Question 4

Describe how information is determined in terms of bits?

Answer 4

A bit is a binary choice to get from one state to another, the more binary choices to get from one state to the other, the more bits it requires.

It takes more binary choices to get from uncertainty to a known state, and fewer binary choices to get from a partially known state to a known state.

Question 5

What does the structure of a distribution with high entropy versus one with low entropy look like?

Answer 5

High entropy is flatter with less structure, low entropy is more structured and less smooth.