Information Theory 2 Flashcards
(19 cards)
If noise equals zero, what is the mutual information between X to-> Y equal to?
I(X,Y) =
if noise=0, then X=Y
thus
I(X,Y)=H(X)=H(Y)
mutual information =entropy of X = entropy of Y
If there is no noise between the information transfer between X to Y, what is the entropy at X and Y?
entropy of X is the same as the entropy at Y
H(X)=H(Y)
What is the equation for mutual information between points X to-> Y?
What is mutual information
mutual info of X and Y = entropy of Y - noise entropy
I(X,Y) = H(Y) – H(Y|X)
how much you can predict Y when you know X
What is the equation for mutual information of a message transferred from points X to -> Y, when there is noise?
What is the mutual information of a message transferred from X to -> Y when there is NO noise? why?
I(X,Y)= H(Y) - H(Y|X)
or I(X,Y)=H(X) - H(X|Y)
mutual info = entropy - noise
(MI is symmetric and can be calculated in different ways)
I(X,Y)=H(X)=H(Y)
because you cancel out the noise variable (+ MI is symmetric)
Is the message identical at both points (X and Y) when there is noise?
no!
no noise = identical
Why is the message at points X and Y not identical when there is noise?
because information is lost when sending message from X to Y due to noise
How do you calculate the noise entropy of X->Y?
noise entropy = H(Y|X)
entropy of Y given X
With a message sent from X to-> Y, why is the noise H(Y|X)?
H(Y|X) means the amount of information received when X is held constant ( when no information sent in channel). thus this means the amount of information in the NOISE transmitted in the channel
How does information theory relate to neurons?
its a neurons job to transmit information from one point to another
H1= a motion-sensitive neuron in the fly’s visual system
What view did the paper by Ruyter van Steveninck et al. challenge?
What was their result and what does this suggest?
How did implement information theory in their experiment?
What equation did they use?
-that the spike train pattern of neurons is random and not stimulus-specific
-spike trains showed remarkably high reproducibility when exposed to natural stimuli -> suggests that the neuron’s spike timing is precise and stimulus-locked, not random (temporally coded)
-reproducibility are quantified with ideas from information theory
high reproducibility -> high information
so when there was high mutual information and less noise, this was driven by a natural stimulus
I(S,R)= H(S) - H(S I R) equation for MI
where S=signal and R=response
What did Strong. et al find out about neuronal responses?
What is the reason for this?
-neurons do not use their full coding capacity = they calculated the maximal entropy (of the neural response) could be 10bits however the actual response was only 5bits (dont maximise entropy)
-energy constraints: high energy cost of spiking
What else did Strong et al. conclude about the neuronal response?
half a neuron’s entropy is signal, the other half noise
What can we use information theory to calculate in neurons?
-the amount of info transferred
-to calculate how precise spiking should be
What did Butts et al. discover about neural stimuli?
How did they implement information theory?
spikes need to be precise on a shorter timescale to transmit information about the stimulus
used IT to calculate how precise spiking should be
What is the issue with these experiments implementing IT?
can only run a low number of repeats -> systematically biases information measures
low number of samples -> data is skewed
What is the issue with having low sample size in IT neuroscience studies?
What does this do to entropy and MI? why?
low sample size biases entropy downwards
therefore MI will be biased upwards
(MI overestimated due to low entropy)
What is the solution to low sample size bias?
extrapolation
Why does a small sample size cause lower entropy?
fewer samples for there to be surprise
more predictable
