Week 9 - Neural Networks

Question 1

What are the two types of learning a neural network can do?

Answer 1

Supervised and unsupervised

Question 2

What is supervised learning?

Answer 2

The output of the neural network is compared against the correct output. The network then corrects itself based on that output. Overall, the data is labelled

Question 3

What is unsupervised learning?

Answer 3

The data is not labelled. The network organises itself according to patterns in the data and no external “desired output” is provided

Question 4

What does a perceptron consist of?

Answer 4

It consists of a set of weighted connections, the neuron (incorporating the activation function) and the output axon

Question 5

How does a perceptron learn?

Answer 5

Initialise weights & threshold

Present the input and desired output

Calculate the actual output of the network:
For each input:
* Multiply the input data (xi) by its weight (wi).
* Sum the weighted inputs and pass through the activation function
Adapt the weights:
* If correct wi(t+1) = wi(t)
* If output 0, should be 1 wi(t+1) = wi(t)+xi(t)
* If output 1, should be 0 wi(t+1) = wi(t)-xi(t)

Question 6

What can the weight update function be used to give?

Answer 6

A decimal term η between 0.0 and 1.0 to slow learning

Question 7

What does the Widrow-Hoff Learning Rule give us?

Answer 7

Weight updates proportionate to the error made, giving Δ = desired output – actual output

Question 8

Name a limitation of the perceptron.

Answer 8

Only linearly separable problems can be solved

Question 9

What does linearly seperable mean?

Answer 9

A straight line which seprates two classes can be drawn

Question 10

What are the three layers of an MLP (Multi-Layer Perceptron)?

Answer 10

Input, hidden, output

Question 11

What are weights in terms of neural networks?

Answer 11

Variable strength connections between units that propagate signals from one unit to the next. They are the main component changed during learning

Question 12

Describe the feedforward learning algorithm.

Answer 12

Initialise weights and thresholds to small random values.

Present input and desired output.

Calculate actual output by:
- Multiplying incoming signal by weight
- Pass this through sigmoid activation function
- Pass on this output to units in the next layer

Question 13

Describe the backpropagation learning algorithm.

Answer 13

Adapt the weights.

Start from the output layer and work backwards:
- New weight (t+1) = old weight, plus a learning rateerror for pattern p on node joutput signal for p on j

Compute error as follows:
- For output units, compute error sigmoid derivativetarget output - actual output
- For hidden units, use the sigmoid derivativeweighted error of the k units in the layer above

Question 14

What are the two types of weight updating?

Answer 14

Batch and online

Question 15

Describe batch weight updating.

Answer 15

All patterns are presented, errors are calculated, then the weights are updated

Question 16

Describe online weight updating.

Answer 16

The weights are updated after the presentation of each pattern

Question 17

What does momentum do in terms of weight updating?

Answer 17

Encourages the network to make large changes to weights if the weight changes are currently large. This allows the network to avoid local minima in early stages as it can overcome hills

Question 18

List some differences between symbolic AI and connectionism?

Answer 18

Symbolic:
explicit representation (symbols)
expert system (IF-THEN) rules
enabling reasoning
knowledge programmed (by humans)
serial (fragile)
does not generalise (outside scope)
understandable/explainable

Connectionism:
implicit representation (numbers)
neural network (weighted graph)
enabling perception
knowledge learned (from data)
distributed (graceful degradation)
generalise (outside scope)
black-box

Question 19

Name some properties of neural networks.

Answer 19

Able to learn to relate input variables to required output

Able to generalise between samples

Shows graceful degradation

Question 20

Define classification.

Answer 20

The output of the function to learn is a class (discrete)

Question 21

Define regression.

Answer 21

The output of the function to learn is a value (continuous)

Question 22

What is graceful degradation?

Answer 22

In symbolic systems, the removal of one component of the system usually results in failure

Removal of neuron(s) from a neural network will reduce performance and probably not result in overall failure, which replicates out understanding of fault tolerance in the brain (graceful degradation)

Question 23

What is classification designed to do?

Answer 23

Group samples according to some known properly

Question 24

What are the two datasets required for a classification problem?

Answer 24

Training (consists of a set of measurements (inputs) and a class (output), used for learning) and testing (“unseen” examples to test generalisation)

Question 25

What is an issue in representing continuous data in a neural network?

Answer 25

Can require normalisation depending on the activation function

Question 26

What is an issue in representing discrete data in a neural network?

Answer 26

Each value needs to have a separate representation in the network to avoid implied order bias (noise that can confuse a neural network)

Question 27

What are the two representations of discrete data in a neural network?

Answer 27

Field-type and thermometer-type

Question 28

When does overfitting occur and how can it be overcome?

Answer 28

When the network is trained on a task for too long, the network learns the noise in the input as well as common patterns, so the result is poor on unseen examples

It can be overcome by stopping the network learning earlier - early stopping

Question 29

What is a common method for early stopping?

Answer 29

Cross-validation - have three data sets: training, testing and cross-validation