Deep Learning - Dr Bashivan

Question 1

why should you not include every detail in a neural model?

Answer 1

1. more difficult to interpret
2. lower feasibility
3. more difficult optimization

Question 2

what is the current practical sweet spot for amount of detail integration?

Answer 2

deep neural nets

Question 3

what is the up and down of verbal explanations?

Answer 3

• easy to communicate!
• has a narrow bandwidth :/

Question 4

what is the up and down of quantitative explanations? (code)

Answer 4

• easily transferrable, easy communication, can answer questions without costly experiments
• requires coding literacu

Question 5

what is the classic approach for studying neuroscience?

Answer 5

identify and characterize individual elemnts in the brain (bottom->up approach)

Question 6

what is the difference between machine learning and deep learning?

Answer 6

machine: figure out a template/feature of what you are looking for and then classify
deep learning: feature extraction + classification happen at the same time

Question 7

why is the classic approach for studying the brain not so efficient?

Answer 7

only considers one of few tasks at a time, and only a few neurons

Question 8

give an example of the classic approach

Answer 8

surround modulation and two-interval discrimination

Question 9

what components is the deep learning framework based on?

Answer 9

• architecture
• learning objective (cost functions)
• learning rule
• dataset (secondary axis)

Question 10

what are 3 principles of holistic deep learning approach?

Answer 10

• units have ubiquitous functionality
• units’ function diversity comes from autonomous learning
• groups of units are orchestrated to facilitate internalized or external objectives

Question 11

name 2 static architecture models

Answer 11

• multilayer perceptrons
• convolutional neural network

Question 12

what is multilayer perceptrons?

Answer 12

each unit in a layer is connected to all the units in the previous and following layer

Question 13

what in convolutional neuron network?

Answer 13

units are locally connected to subgroups of units

Question 14

name the 2 dynamic architecture models

Answer 14

• recurrent neural network
• transformers

Question 15

what is recurrent neural network?

Answer 15

internal memory gets updated based on observations

Question 16

what are the 3 types of cost-functions strategies?

Answer 16

unsupervised, supervised, reward-based

Question 17

what is unsupervised objective (cost) functions?

Answer 17

• learn from observations, model reproduces what it sees: predicting errors, continuity, sparsity
• has generative consistency: wake-sleep algorithm, generative neural networks

Question 18

give an example (allegory?) of unsupervised objective functions

Answer 18

finishing someone’s sentence

Question 19

what is a downside of unsupervised learning algorithms?

Answer 19

it may fail to discover properties of the world that ae statistically weak but important for survival

Question 20

how can we solve the problem of unsupervised objective functions not discovering essential properties?

Answer 20

supervised objective functions

Question 21

give examples of supervised objective functions

Answer 21

object recognition
object detection
source localization

Question 22

what is reward-based cost functions?

Answer 22

agents try to maximize reward

Question 23

how are costs encoded in the brain vs in neural net?

Answer 23

brain : genes
neural net:
- cost-encoding neural net (small)
- task-performing neural net (large)

Question 24

what are the 3 learning rules you can use?

Answer 24

following a gradient, not following a gradient, partially following a gradient

Question 25

why do we think prefrontal cortex neurons continue to fire during the delay period despite no stimulus?

Answer 25

to keep the novel information in mind

Question 26

what did they find after making monkeys perform an oculomotor delayed response task?

Answer 26

during the delay period, neurons fire or are inhibited selectively to the cue location

Question 27

what fraction of PFC neurons shows excitation or inhibition during the delay?

Answer 27

1/3

Question 28

name the 4 toolkit for model testing

Answer 28

1. behavioural agreement 2. agreement with neural data 3. in silico electrophysiology 4. developmental agreement

Question 29

what is behavioral agreement toolkit?

Answer 29

quantifying the behavioral similarity of our network vs animal network doing the same task

Question 30

what is agreement with neural data toolkit?

Answer 30

comparing how the model vs animal solves the task

Question 31

what are 2 different ways of testing agreement with neural data?

Answer 31

1. representational similarity analysis: comparing patterns of responses using matrix 2. encoding model: compare neuron with a unit

Question 32

what are the 3 types of in silico electrophysiology

Answer 32

- lesion studies - decoding (find how a characteristic of the task is encoded) - selectivity profile

Question 33

what is developmental agreement toolkit?

Answer 33

performing previous analyses at different stages of learning

Question 34

explain the artificial neuron model

Answer 34

each dendrite works as an input channel -> some weight each input and assignes a value to each -> if weighted sum reaches threshold the neuron starts spiking

Question 35

what was the first neural net?

Answer 35

multi-layer perceptron: extended artificial neuron model into interconnected layers of neurons

Question 36

what was the limitation of the multi-layer perceptron?

Answer 36

it only considers a limited part of the visual field (units are only connected to the units around the center of the previous layer)

Question 37

convolution neural networks allow...

Answer 37

Patterns to be distinguished regardless of their spatial location

Question 38

what is convolutional neural networks?

Answer 38

network learns and applies a quernel / convolution with specific feature and recognizes them in the environment

Question 39

what is alexnet?

Answer 39

convolutional neural network with 9 layers of convolution, pooling, nonlinearity, normalization

Question 40

how was alexnet trained?

Answer 40

supervised training with imagenet dataset

Question 41

what did they find in the first layer of alexnet?

Answer 41

patterns useful for image recognition, similar to those found in V1

Question 42

what did they use the representation dissimilarity matrix for?

Answer 42

to compare the response of neurons of V4, IT, and CNN last layer to 8 different categories of objects

Question 43

the 3rd and 4th layers of alexnet CNN corresponded to what macaque brain areas?

Answer 43

3rd layer = V4 4th layer = IT

Question 44

name 3 ways how alexnet was doing unsupervised learning

Answer 44

1. deep cluster: groups its inputs in clusters 2. instance discrimination: discriminate between pairs of observations from memory 3. contrastive learning: learns to respond similarly to different variations of the same observation

Question 45

do neural networks have a spatial map?

Answer 45

no, selectivity of the units is completely random. no topography

Question 46

how large the weight is between 2 units is scaled by what?

Answer 46

the physical distance between those 2 units

Question 47

as you go higher in the visual pathway hierarchy, what happens to topography?

Answer 47

increased topography organized by categories (more organization)

Question 48

what are the 2 topographic models propose (models for learning objective)?

Answer 48

1. wiring cost minimization (distance between neurons) 2. spatial cost function (spreading of neurons)

Question 49

what does the correlation between response similarity and cortical distance show?

Answer 49

spatial loss hypothesis encourages local correlations