[2] Neural Networks

Question 1

What determines the output of a neuron?

Answer 1

It is the biased, weight sum of its inputs passed into an activation function

Question 2

Why are activation functions important?

Answer 2

They allow the network to learn non-linearities

Question 3

What is a perceptron?

Answer 3

A special type of ANN with:

• Real-valued inputs
• Binary output
• Threshold activation function

Question 4

How are perceptrons trained?

Answer 4

Increase the weights (this is the threshold) based on whether the class is higher or lower than the perceptron

Question 5

What idea limits the generalisability of perceptrons?

Answer 5

The Perceptron Convergence Theorem stats that perceptrons will converge if and only if the problem is linearly separable

Hence, they can’t learn XOR

Question 6

What are the general approaches to updating weights?

Answer 6

Online learning updates weights after every instance; offline learning does it after every epoch.

Batch learning updates weights after every batch of instances

Question 7

What algorithm is used to train neural networks?

Answer 7

Backpropagation:
[1] Calculate the predicted output using the current weights
[2] Calculate the error
[3] Update each weight in proportion to its gradient to the error i.e. how much changing that weight affects the error

Note: weights are trained backwards i.e. start at the last hidden layer

Question 8

What are some potential issues when using backpropagation?

Answer 8

Improper learning rate leads to divergence or slow convergence

Overfitting if training too long, for using too many weights, or using too few instances

Local minima

Question 9

How should variables be represented in an ANN?

Answer 9

Use a binary representation (i.e. one hot encoding) for nominal variables

For numeric variables, consider scaling or standardisation

Question 10

What is scaling and standardization? When should each be used?

Answer 10

Scaling - scale then numbers between [0,1] if they are on a similar range

Standardisation - assume a normal distribution and scale it to N(0,1) if the values are more varied

Question 11

What can happen if ANN weights aren’t set appropriately?

Answer 11

If they are all set to 0, the network will be symmetric i.e. all the weights will change together, and so it won’t train

If the weights are too high, the activation will be in the part of the sigmoid with a shallow gradient, and so training will be slow,

Question 12

How should ANN weights be set?

Answer 12

Using fan-in factor, i.e. using a uniform random generator between -1/sqrt(d) and 1/sqrt(d) where d is the number of inputs

This ensures the variance of the weighted sum is approximately 1/3

Question 13

How can back propagation be sped up?

Answer 13

With momentum, in which gradients from previous steps are used in addition to the current gradient

Question 14

How can weight matricies be visualised?

Answer 14

With Hinton diagrams, in which the size of the square is based on the magnitude; it is white if it is positive and black if it is negative

Question 15

What are the key principles of CNNs?

Answer 15

The automatically extract features to produce a feature map

They are not fully connected - convolutions with shared weights are used instead

Question 16

What are the dimensions of a feature map?

Answer 16

In each direction, it is (image_size - filter_size) / shift + 1

Question 17

What techniques can be applied to optimise CNNs?

Answer 17

• Subsampling aggregates based on the maximal value; this reduces data while retaining/emphasizing the information
• Weight smoothing use used when domain specific knowledge suggests adjacent inputs are related
• Centered weight initialization starts with higher weights in the center, as these are often where objects are found

Question 18

What is a weight agnostic network?

Answer 18

It has a single weight shared by the whole network; training is done by newtwork topology search.

Question 19

What operations occur while training a weight agnostic network?

Answer 19

• Insert node by splitting an existing connection
• Add connection - connect two previously connected nodes
• Change activation - change the activation function of a node

Question 20

What are HONNs?

Answer 20

Higher order neural networks connect each input to multiple nodes in the first hidden layer.

The order is the number of nodes that each input connects to.

CNN are a special type of HONN

Question 21

Why are HONNs useful?

Answer 21

Instead of just taking the weighted sum, for each combination of inputs they take a sum of the weighted products.

This allows them to explore higher order relationships i.e. products; for example, they can solve XOR

Question 22

What are self-organizing maps?

Answer 22

They represent high dimensional data in lower dimensions by weighting mapping inputs to neurons

The weights are trained by competitive learning in which the node whose weight is closest to the input value is chosen to fire. It updates its weights to reinforce those that made it win. the neighborhood function preserves topology

Question 23

What are residual neural networks?

Answer 23

They have shortcut connections between layers.

This makes training more effective as it reduces the vanishing gradient effect

Question 24

What is EvoCNN?

Answer 24

A GA to automatically train network structures. It uses a two-level encoding to describe the layers and then their connections

Each mutation performs one of three actions:

• Add a new unit (convolutional, pooling or full)
• Modify an existing unit’s encoded information
• Delete an existing unit

Question 25

What are auto-encoders?

Answer 25

Neural networks that have been trained to copy their input to their output

The use an intermediate layer called the latent representation

Question 26

How is the loss of an auto-encoder calculated?

Answer 26

The difference between the input and output (or a special domain-specific variations)

Question 27

What are the main configurations of auto encoder?

Answer 27

Under-complete auto encoders have latent representations with smaller dimensions than the input and output.

Otherwise, they are over-complete

Question 28

What do under-complete auto-encoders do?

Answer 28

They learn the most salient features of the data

Question 29

How do over-complete auto-encoders work?

Answer 29

They use regularisation to avoid simply copying the data

Sparsity auto encoders trie to push as many output values of the latent representation to 0 as possible

Contractive auto encoders regularise by derivative penalty, meaning the output of each node is smooth if the input changes slightly. This makes them robust to slight fluctuations

Question 30

What are some particular applications of auto encoders?

Answer 30

De-noising auto-encoders remove noise from the image

Variational auto-encoders modify an image in a desired way. However, the latency space must might not be continuous and so instead of a point in the latent space, a distribution is used.

Question 31

Why is cross-entropy used?

Answer 31

It’s gradients are more pronounced at extreme values, leading to faster convergence

Question 32

Why is ReLU often used?

Answer 32

It is fast to compute, minimises the impact of vanishing gradient, and encourages sparsity

Question 33

What is the purpose of regularisation?

Answer 33

It prevents weights from getting too large, and pushes as many to zero as possible (allowing them to be ignored)

Question 34

What is a particular type of regularisation?

Answer 34

Lasso regularisation uses L1 to remove irrelevant variables from a linear model

Question 35

How does dropout work?

Answer 35

A random percentage of neutrons are removed on each mini-batch

Note: for inference, the weights must be multiplied by (1-p)%

Question 36

What are the general strategies for transfer learning?

Answer 36

Learn shared hidden representations (i.e. DLID). This is useful if the classes are the same, but the way they are captured differs (i.e. different camera types)

Shared features - use this when the head layers do the same general task, but the tail does a particular task.