Week 6 - Deep Learning & CNNs

Question 1

What are the differences between MLPs and CNNs?

Answer 1

MLP:
- Wide application scenario, not just images
- Neurons are fully connected, but can’t scale well to large size data such as images

CNN:
- Neurons are arranged in ‘3D’, each neuron is only connected to a small region of a previous layer
- Typical CNN structure: Input-Convolution Layer-Activation Layer-Pool Layer-Fully Connected layer-Output Layer

Question 2

What are the four parameters in a Convolution Layer?

Answer 2

• Input size (W)
• Filter Size (F)
• Zero Padding (P)
• Stride (S)

Question 3

What is the equation used to calculate the output volume size from a Convolution Layer?

Answer 3

(W - F + 2P) / S + 1

Question 4

What is the primary aim of a Convolution Layer?

Answer 4

Convolution layers are filter banks performing convolutions with the learned kernels/masks.

Question 5

What is the main purpose of a Pooling layer?

Answer 5

A Pooling layer aims to pool the results from many convolution layers, which reduces the resolution of the filter outputs. As such, subsequent convolutional layers therefore access larger areas of the image.

Question 6

What is the effect of the Pooling Layer?

Answer 6

It reduces the spatial size of the representation and reduces the amount of parameters, effectively down-sampling the input to increase the receptive field size.

Question 7

What are the four types of activation functions?

Answer 7

• Sigmoid
• Tanh
• ReLU
• Leaky ReLU

Question 8

What are the properties of Sigmoid?

Answer 8

• Range of values is [0, 1]
• Saturates and kill gradients, with small gradients at regions of 0 and 1

Question 9

What are the properties of Tanh?

Answer 9

• Has a zero-centred range between [0, 1]
• Saturates and kill gradients, with small gradients at regions of 0 and 1

Question 10

What are the properties of ReLU?

Answer 10

• Solves vanishing/exploding gradient problems
• Simple to calculate
• Some neurons can be ‘dead’ with negative input

Question 11

What are the properties of Leaky ReLU?

Answer 11

• Overcomes the dying neuron problem
• Performance is not consistent

Question 12

What is the function of the Fully Connected Layer with Softmax?

Answer 12

Fully Connected Layer is fully connected to all activations in the previous layer
The Softmax function converts the prediction range to the range of [0, 1] for each class

Question 13

How does a Softmax function work?

Answer 13

A softmax function takes a set of numbers as an input, and outputs a probability distribution spread over a set of k classes., which gives us a probability for each class in the range of [0, 1]

Question 14

What is a key property of the Softmax function?

Answer 14

The probabilities produced all sum to 1 across the classes. Example:
Class 1 = 0.4, Class 2 and 3 = 0.3
Therefore 0.4 + 0.3 + 0.3 = 1

Question 15

What is the typical architecture for CNNs in Classification problems?

Answer 15

Typical block:
- Normalisation Layer
- Filter Bank
- Non-linear
- Feature Pooling

After multiple blocks of the above:
- Classifier

Question 16

What makes a CNN effective?

Answer 16

• Increased depth
• Smaller filters at the lower levels
• More convolutional than fully connected layers

Question 17

What is a hierarchical model in terms of CNNs?

Answer 17

A hierarchical model is a type of CNN which learns details at the pixel level, and then with each block ‘zooms out’ and finds less ‘zoomed in’ details. Example:
First stage looks at pixel-level details, then the second stage looks at edges. Third stage then looks at object parts i.e. combination of edges, before the final stage looks at object models