Week 3: Introduction to Neural Networks

Question 1

Neuron

Answer 1

An information processing unit.

Question 2

Action potential

Answer 2

The signal outputted by a biological neuron.

Question 3

Firing rate

Answer 3

The number of action potentials emitted during a defined time-period.

Question 4

Synapse

Answer 4

The connection between two neurons.

Question 5

Artificial Neural Network

Answer 5

A parallel architecture composed on many simple processing elements interconnected to achieve certain collective computational capabilities.

Question 6

De-noising Autoencoder

Answer 6

The network is trained so that the output, r, reconstructs the input, x. However, before encoding is performed the input is corrupted with noise. This mitigates overfitting.

Question 7

Processing Units

Answer 7

For neural networks, they’re organised into layers and each have an activation function, as well as weights for each neuron.

Question 8

Perceptron

Answer 8

A linear threshold unit.

Question 9

Logical Functions

Answer 9

Can be AND, OR, XOR, or other arbitrary logical functions. Not all logical functions are linearly separable.

Question 10

Sequential (Online) Delta Learning Algorithm

Answer 10

For each sample, x_k, in the dataset in turn, w <- w + eta(t_k - H(wx_k))x_k^t. Do until the algorithm converges. One update of the parameters is based on 1 sample. The order in which the samples are used may affect the speed of convergence. It doesn’t necessarily outperform batch learning. This approach only depends on misclassified samples.

Question 11

Batch Delta Learning Algorithm

Answer 11

For each iteration, go through all the all the samples, calcualte eta(t-y)x^t, then sum up the results. That’s the update for the weights. Keep iterating until the weights are unchanged. One update of the parameters is based on n samples. Sample order isn’t relevant in batch learning. This method only depends on misclassified samples. Batch learning can be faster than Sequential learning, because it processes the entire batch at once using vectorisation, which allows the parallel computing of each sample in the batch.

Question 12

Hebbian Learning Rule

Answer 12

An unsupervised method which strengthens connections between active neuron and any active inputs. Sequential (Online) Learning Algorithm is w <- w + eta y x^t

Question 13

Competitive Learning Algorithms

Answer 13

When output units compete for the right to respond to input, meaning that some neurons have their activity suppressed by other neurons.

Question 14

Negative Feedback Networks

Answer 14

When output units compete to receive inputs, rather than compete to produce output.

Question 15

Autoencoder Networks

Answer 15

When the output, r, is a reconstruction of the input using a separate neural population, with inhibitory feedback connections removed, and neural responses no longer having to be calculated iteratively. It can still learn weights as to minimise error between input and reconstruction.

Question 16

Neural Networks

Answer 16

Feedforward Networks: Single-layer Perceptron, Multilayer Perceptrons, Radial Basis Function Nets. Feedback/Recurrent Networks: Competitive Networks, Kohonen’s SOM (not covered), Hopfield Network (not covered), Negative Feedback. For classification tasks, there should be one neuron per class in the output.

Question 17

Mini-batch Delta Learning Algorithm

Answer 17

This method divides the samples into smaller batches and trains batch-by-batch. This method can optimally exploit the tradeoff between the vectorisation speed up from batch learning, the more frequent updates from sequential learning to increase accuracy in the estimation of the gradient.

Question 18

Softmax

Answer 18

A popular activation function used in output layers in classification tasks. y_i \leftarrow \frac{e^{\beta y_j}}{\sum_k e^{\beta j_k}}