Past Exam Questions

Question 1

Q1. What is the analogy in Artificial Neural Networks to synaptic efficiencies in biological
neurons?
(2p)

Answer 1

• The strength of the weights between neurons
• They can be updated (strengthened or weakened) depending on learning performance
  (as for synaptic transmission)

Question 2

Q4. Which of the following non-derivative functions can only produce outputs in the range 0 to
1? [may be more than one correct answer]

a. Linear activation?

b. The rectified linear unit (ReLU)
activation function?

c. The sigmoid function?

d. The softmax function?

Answer 2

c. and d. (These two functions are squashing functions, softmax outputs must add to 1). Linear
activation produces output that is a linear function of the input (not squashed). ReLU is a linear
activation function for inputs of >=0 and outputs 0 otherwise. Note, you are not required to
explain here though, just give the letters c. and d.)

Question 3

Q5. How many weights (excluding threshold/bias) does a feedforward, fully-connected, neural
network with a single hidden layer and with 5 inputs nodes, 3 hidden nodes and 1 output node
and no biases have? [only one correct answer]
a. 13
b. 18
c. 9
d. 20

Answer 3

Answer: b.

Question 4

Q8. Describe the difference between supervised and unsupervised learning.

Answer 4

Answer:
- Supervised learning entails the ANN learning to predict specific outputs (targets)
labelled by humans on a classification task. E.g. dog images are labelled as dogs and
the ANN has to learn that the image is a dog rather than a cat).
- Unsupervised learning entails the ANN learning without specific human-labelled
targets. E.g. unsupervised autoencoders learn according to how well they re-present
their input – this is a sort of ‘target’ but it does not involve human labelling.

Question 5

Q1. Which of the following, given appropriate connection weights, can a Multi-Layer
Perceptron with one hidden layer find linearly-separable classifications for its inputs? [may be
more than one correct answer]

a. AND logic functions?

b. OR logic functions?

c. XOR logic functions?

Answer 5

Answer: a. b. and c. (MLPs can solve all such Boolean logic functions including Not-AND, NotOR, Not-XOR, which give the opposite outputs).

Question 6

Q1. In what ways can Convolutional Neural Networks be seen as similar to how the Visual
Cortex processes stimuli?
(3p)

Answer 6

• Cells in V1 are receptive to simple features (as can be seen in early convolutional
  layers with relatively small filter sizes)
• Cells in Visual cortex also appear to encode for pooling.
• Cells in higher levels of visual cortex and Inferotemporal cortex are responsive to
  objects and relatively invariant to the spatial and orientation presentation of the
  object.

Question 7

Q2. How many pooling maps does a standard ‘shallow’ Convolutional Neural Network (1 layer
of feature maps + 1 layer of pooling maps) have that uses 5 filters of dimensionality 4x4 over
the input image?
a. 5
b. 10
c. 16
d. 32

Answer 7

Answer. a. (each filter is used to encode features on a single feature map, each feature map
outputs to a single pooling map, size of the filter here is irrelevant).

Question 8

Q4. Describe how Convolutional Neural Networks encode for spatial invariance when being
trained on multiple images of the same class.

Answer 8

Answer:
- CNNs use pooling maps that encode a lower dimensional ‘summary’ of the space of
the features maps to which they respond by encoding, for example, the maximum
value in an area of space on the feature map. This means that if the object in the
image shifts (relative to other images of the same class) but is still in the receptive
field of the pooling map neuron, the neuron will still be responsive.
- Because deeper CNNs have increasingly lower dimensionality (fewer neurons) in
deeper layers, the representation of space becomes increasingly compressed, the
receptive field becomes larger, and neurons (especially in pooling regions) become
increasingly invariant to shifts of the object in the image.
- If the pooling neuron’s receptive field covers the entire image, it should be invariant
to any shift of the object in the image space.
(note, this is quite a detailed answer, it could be possible to score 3p with a slightly less detailed
answer).

Question 9

Q5. Describe two ways in which Deep Convolutional Neural Networks allow for a small number
of weights/parameters with respect to MLPs like stacked autoencoders.
(4p)

Answer 9

Answer:
- They use low-dimensional filters over the images they are processing and these filters
(vector of weights) are shared for the feature map that is encoding the same feature
in different positions in space.
- A large number of weights (usually most of the weights) are found between the final
pooling layer and the fully connected layers. The deeper the network, the lower the
dimensionality of the network. So, a deeper network with low dimensionality pooling
maps will have fewer connections to the fully connected layers as compared to a
shallow network with higher dimensionality. So, as a result, there may be fewer
weights over the whole network when adding layers. This is not true for a stacked
autoencoder that is fully connected all the way through and so more layers necessarily
means more weights.
(Note, the two points should be reasonably detailed to get the 4p).

Question 10

Q6. Give three reasons why deeper Convolutional Neural Networks might be considered
superior to more shallow Convolutional Neural Networks.
(3p)

Answer 10

Answer:
- Training efficiency: Greater depth means fewer parameters between the final pooling
layer and the fully connected layers, i.e. so doesn’t necessarily increase the number of
parameters overall (see Q5).
- Spatial invariance: Greater depth increases the receptive field of the network (large in
later layers) and therefore allows for more of the image space to be covered.
- Feature combination: Greater depth allows for more possible combinations of
features for reconstructing/representing the images in greater detail.