Week 5

Question 1

What is a deep NN? What is shallow NN?

Answer 1

Deep is > 3, typically&raquo_space; 3 layers

Shallow is <=3 layers

Question 2

Why question he’s of Deep NNs? Why should we use?

Answer 2

Shallow NN can appreciate any continuous non linear function arbitrarily well

Deep helps capture complexities

Question 3

Unnecessarily confusing example of RNN

Answer 3

Because output of last hidden layer goes back into input of first hidden layer, in this example, hidden layer goes into itself

Question 4

Difficulties with backprop

Answer 4

Vanishing and exploding gradients

Question 5

Mitigate vanishing gradients

Answer 5

Use activation functions with non vanishing gradients

Question 6

Mitigate exploding gradients

Answer 6

Batch normalisation

Question 7

Activation functions with non vanishing derivatives

Answer 7

Question 8

Better ways to initialise weights

Answer 8

Question 9

How does back prop struggle with gradients in cost function

Answer 9

Variation in magnitude of gradient may occur between:
- different layers
- different parts of the cost function for a single neuron
- Different direction for a multi dimensional function

Question 10

Momentum in back prop ? Useful for?

Answer 10

Adds moving average of previous gradients to current gradient (helps with plateaus and local minima)

Movement = negative of gradient + momentum

Question 11

Adaptive learning rate and examples

Answer 11

Vary learning rate (for individual parameters) during training:
- increasing learning rate if cost is decreasing
- decreasing learning rate if cost is increasing

Eg AdaGrad or RMS Prop

Question 13

Examples of back prop algorithms with adaptive learning rate and momentum

Answer 13

ADAM
Nadam

Question 14

Motivate batch Normalisation

Answer 14

Different inputs to 1 neuron can have very different scales (depending on the neuron the input comes from)

Question 15

Batch normalisation

Answer 15

Question 16

Benefit of batch normalisation

Answer 16

Question 17

Skip connections

Answer 17

Question 18

Define CNN and motivate

Answer 18

Any NN in which at least 1 layer has a transfer function implemented using convolution/crosscorrelation

Motivated by desire to recognise patterns with tolerance to location

Question 19

Weight sharing in CNNs, how?

Answer 19

Question 20

Transfer function for CNN

Answer 20

Question 21

Output of 1 neuron of CNN layer

Answer 21

Array of numbers, sometimes called feature map

Question 22

Cross correlation formula

Answer 22

Question 23

Convolution formula (the operation)

Answer 23

Question 24

Mask for CNN

Answer 24

Question 25

Variations of standard cross correlation

Answer 25

Question 26

Multiple masks

Answer 26

Question 27

How to reduce number of channels and computational burden of next layer

Answer 27

use 1x 1 convolution

Question 28

Reasons for pooling layers in CNNs

Answer 28

To increase tolerance to: Location of patterns Configuration of sub patterns

Question 30

How pooling layers work

Answer 30

Replacing values within a region of the input array with a single value calculated (eg mean “average pooling”, or maximum “max pooling”)

Question 31

Pooling typically has

Answer 31

Stride > 1 to decrease spatial size of the layer

Question 32

Different sized pooling regions can be used if

Answer 32

Pooling regions here are RFs Used if RF is same size as input array, output is scalar value “global pooling”

Question 33

Is pooling operation learnt

Answer 33

No, user specified

Question 34

Other names for pooling

Answer 34

Sub sampling Down sampling

Question 35

Typically pooling is applied separately to each input array, therefore

Answer 35

The number of output channels is equal to the number of input channels (However it is possible to perform cross channel pooling to reduce the resulting number of channels)

Question 36

Final layers of CNNs are typically? Especially when?

Answer 36

Typically fully connected Especially when applied to classification tasks

Question 37

To create the input to the 1st fully connected layer (for CNN) it is necessary to

Answer 37

Flatten the output of the last convolutional or pooling layer

Question 38

3 issues with Deep NNs and why

Answer 38

Deep NNs/CNNs have lots of tunable parameters Therefore: 1) need lots of training data 2) training is computationally intensive 3) there is a danger of over fitting on training data

Question 39

How to address need for lots of training data for deep NNs/CNNs

Answer 39

Data augmentation (eg applying transformations to images) Transfer learning

Question 40

How to address overfitting of CNNs/Deep NNs

Answer 40

Regularization

Question 41

Dropout regularization

Answer 41

During usage, all neurons behave normally Fro put forces random sub networks to correctly classify each sample, preventing recognition of individual samples being reliant on individual neurons

Question 42

Even if all precautions taken; what issue can occur with training deep NNs/CNNs

Answer 42

Failure to generalise