week 3 - Efficient Coding I Flashcards
(23 cards)
What is R replaced for in the efficient coding hypothesis?
In the EC hypothesis:
What does R represent?
What does S represent?
R = f(S) the tuning curves f to be optimized
EC hypothesis = Maximize I(S;R) = Maximize I(S;f(s))
S= optimal signal
R= response curve
What are the two parts of the efficient coding hypothesis?
-a group of neurons should -> ENcode as much information as possible
“ -> remove as much redundancy as possible
what is the equation for EC?
Maximize I(S;R) = Maximize I(S;f(s))
where
I = mutual information (between stimulus and neural encoding)
S = signal (optimal signal)
R = the function (tuning curve to be optimised) / neural encoding
Why are natural images assigned a high probability in our own natural image space?
because natural images follow spatial and structural trends which we assign as predictable
Why are non-natural images assigned a low probability in our own natural image space?
non-natural images have lots of random noise and uncorrelated pixels etc making them harder to predict and store
What is the chance of generating a natural image in a image space (not in our brains)? Why?
very small probability that you will get a natural image - the vast majority of images are just noise
To generate a natural image from an image space is it better to use higher order statistics?
best to use only up to second order as it gets too complicated because information processing would take really long as there would be increasing amount of data
In a NATURAL image, are pixels independent of each other?
Which statistical process treats pixels as independent from each other?
no there is a correlation between pixels
-first order statistics
What explains why pixels are not independent of each other?
- if you delete eg 1% of pixels then eg 40% you can still kind of see the picture -> thus pixels aren’t independent
What does second order statistics tell you about pixel correlation in natural images?
pixels are NOT independent of each other:
x and x+1 (pixel adjacent) -> strong positive correlation
x and x+2 -> positive correlation but less strong
x and x+3 -> even weaker pos correlation
What is the caveat of using 2nd order statistics to generate natural images? eg Gaussian Image Model
This is a GOOD method apart from at borders between light and dark (contrast): between dark and light e.g. dark tree stump and white sky background
Is second or first order statistics better at generating natural images?
second order
How does the Fourier domain relate to natural image generation?
fourier decomposition: a negative correlation between
High freq when pixels are changing in brightness very rapidly (as you move from pixel to pixel)
X axis = frequency
Y axis = how often they occur in natural images = power
Natural images have slow changes/slow frequencies
This curve is called = POWER LAW . This law shows up in nature a lot.
Which probability distribution is used to help generate natural images?
How does the distribution do this?
Gaussian/Normal
by capturing the correlations between neighbouring pixels in natural images
What sort of model is the Gaussian Image Model?
What does it model?
Is it 1st or 2nd order statistics?
-probabilistic model
-the correlation between neighbouring images
-2nd order
What is the simplified equation for Gaussian image model?
What does each term mean?
How they model pixel correlations in natural image generation?
p(x)=N(x|μ, ∑)
where
N = normal dist
x = pixel
μ = vector containing mean pixel intensities
∑ = covariance matrix describing correlations between pairs of pixels
image patches (x) tend to cluster around a typical mean shape (μ) and show predictable co-variation (Σ).
What is the issue with the images a Gaussian Image Model generate?
cannot model edges due to its Gaussian assumption (edges a part of natural images)
(nightmare image)
What shape does a Gaussian image model in a graph of pixel brightness of neighbouring pixels x1 (xaxis) and x2 (yaxis)?
elipse which is orientated in to show positive correlation on graph
In the Gaussian Image Model, how do we maximize information when generating natural images?
What is the issue of this from a EC hypothesis pov?
by assuming that natural stimuli come from this simple distribution (Gaussian)
simple method maximises information
you must also remove redundancy/correlation between pixels
Implementing the EC hypothesis, how do we maximize information from a LINEAR Gaussian image model?
information measures simplify when using a simple normal dist:
Entropy -> Variance
Redundancy -> Correlation
thus we must maximize variance & minimise correlation to achieve EC hypothesis
In a linear Gaussian image model, when entropy is simplified to variance, is a fat or a skinny bell curve of a normal distribution better for efficient coding?
FAT:
high uncertainty, high information, high entropy -> high variance (thus maximizing information and better for efficient coding hypothesis)
SKINNY:
vice versa
In EC, why do we want to minimise the correlation between neurons in a group of neurons?
redundancy -> correlation thus would be good for efficient coding
Overall, what would be the most efficient strategy to maximize information in a LINEAR Gaussian image model? why?
Take correlated pixel inputs, and transform these inputs into DEcorrelated neural activity
because decorrelation removes redundancy =efficient coding
