DL CNNs & RNNs

Question 1

How to calculate number of parameters (weights and biases) in a CNN?

Answer 1

((filter width * filter height) * (num of old channels/filters) + 1 for bias) * num of new filters

Question 2

What is dropout?

Answer 2

Randomly setting a fraction of input units to 0 at each update during training time - helps prevent overfitting

Question 3

How do we combat exploding and vanishing gradients?

Answer 3

1. Normaliztion of inputs + Careful initialization of weights
2. Regularization
3. Gradient clipping

Question 4

Tips for Choosing Initial weights

Answer 4

1. Never set to all zero
2. Try somewhere between -0.2 and 0.2 (or fanin)
3. Biases are often initialized to 0.01 or similar small value

Question 5

What is regularization?

Answer 5

“Regularization is any modification we make to a
learning algorithm that is intended to reduce its
generalization error but not its training error.”
Goodfellow et al, 2016

Question 6

Regularization methods for regression

Answer 6

1. Lasoo - L1 encourages sparseness in weight matrices
2. ridge - L2 (weight decay/parameter shrinkage)
3. elastic net - combines lasoo and ridge

Question 7

Inverted dropout

Answer 7

During training randomly drop out units according

to a dropout probability at each training epoch

Question 8

How does dropout work?

Answer 8

Spread out weights - cannot rely on any one input too much

Question 9

Disadv. of dropout

Answer 9

Introduces another hyperparameter - dropout probability - often one for each layer

Question 10

What’s another type of regularization apart from inverted dropout?

Answer 10

• Dataset augmentation
• Synthesize examples by flipping, rotating, cropping, distorting
• makes dataset more robust

Question 11

What is early stopping?

Answer 11

Allowing a model to overfit and then rollback to the point at which the error curve on the training and test sets begin to diverge

Question 12

1:1 RNN

Answer 12

Vanilla network without RNN

Image classification

Question 13

1:M RNN

Answer 13

Image captionning

Question 14

M:1 RNN

Answer 14

Sentiment analysis

Question 15

M:M RNN

Answer 15

Machine translation

Question 16

M:M RNN

Answer 16

Video classification (syncned sequence input and output)

Question 17

Why Convolutions?

Answer 17

The main advantages of using convolutions are parameter sharing and sparsity of connections. Parameter sharing is helpful because reduces the number of weight parameters is one layer without losing accuracy. Additionally, the convolution operation breaks down the input features into a smaller feature space, so that each output value depends on a small number of inputs and can be quickly adjusted.

Question 18

CNNs

Answer 18

Designed to process data that come in the form of multiple arrays, for example, colour images composed of 3 2-D arrays containing pixel intensities in the 3 colour channels

Question 19

Key features of CNNs

Answer 19

• local connections
• shared weights
• pooling
• use of many layers
• roots in neocognition

Question 20

Role of convolutional layer

Answer 20

• detect local conjunctions of features from the previous layer

Question 21

Role of pooling layer

Answer 21

To merge semantically similar features into one

Question 22

Success of CNNs

Answer 22

ImageNet 2012

Halved error rates of competing approaches
Efficient use of GPUs, ReLUs
New regularizations - dropout
Techniques to generate more training examples by deforming existing ones

Question 23

RNNs are good for what tasks

Answer 23

Those that involve sequential input (speech and language)

• process an input sequence one element at a time, maintaining in their hidden units a ‘state vector’ that implicitly contains information about the history of all the past elements of the sequence

Question 24

Useful things about CNNs

Answer 24

1. Partial connectivity (i.e. sparse connections) - not all the units in layer i are connected to all the units in layer i + 1
2. Weight sharing - different parts of the network are forced to use the same weights

Question 25

Four key ideas behind CNNs that take adv of the properties of natural signals

Answer 25

1. local connections 2. shared weights 3. pooling 3. use of many layers

Question 26

Convolutional layer

Answer 26

- units in a conv layer are organized in feature map, within which each unit is connected to local patches in the feature maps of the previous layer through a set of weights called a filter bank - result of this local weighted sum is then passed through a non-linearity such as a Relu - All units in a feature map share the same filter bank - Different feature maps in a layer use different filter banks Why this architecture: 1. Local groups of values are often highly correlated, forming distinctive motifs that are easily detected 2. local statistics of images are invariant to location (a motif can appear anywhere on the image - hence the idea that units at different locations share the same weights Mathematically, the filtering operation performed by a feature map is a discrete convolution (name)

Question 27

Role of conv layer

Answer 27

To detect local conjunctions of features from the previous layer

Question 28

Role of pooling

Answer 28

To merge semantically similar features into one Reduces dimensions of the representation and creates an invariance to small shifts and distortions

Question 29

How is training in a CNN done?

Answer 29

Backpropagating is the same Allows all the filter banks to be trained

Question 30

Deep NNs exploit the property that many natural signals are compositional hierarchies in which higher-level features are obtained by composing lower-level ones.

Answer 30

Images -> local combos of edges form motifs -> parts -> objects Same with speech Pooling allows representations to vary very little when elements in the previous layer vary in position and appearance

Question 31

Issue of representation

Answer 31

debate between logic-inspired and NN-inspired paradigms logic - symbols are identical or non-identical to other symbols, no reasoning, rule-based inference NN - just uses big activity vectors, big weight matrics and scalar non-linearities to perform the type of fast 'intuitive' inferene that underpins effortless commonsense reasoning

Question 32

LSTM

Answer 32

RNNs have difficult holding onto information for very long memory cells - account for explicit memory = special hidden units, accumulator or gated leaky neuron

Question 33

RNNs unfolded

Answer 33

Very deep FF networks - in which all the layers share the same weights

Question 34

CNNs can make use of RL to focus on where to look

Answer 34

attention