Lecture 2

Question 1

What are feed-forward networks?

Answer 1

A “feed forward network” in machine learning refers to a type of artificial neural network where data flows in only one direction, from the input layer through any hidden layers, and finally to the output layer, without any information looping back or being reused from previous layers.

Question 2

What can we say about neural networks with one hidden layer?

Answer 2

If there are enough hidden units, they can approximate any function arbitrarily accurate.

Question 3

What is the proof for saying a neural network with one hidden layer is a universal approximation?

Answer 3

Hand-waving proof rather than mathematical proof (looked at regression).

We can approximate a curve with lots of step functions.

Question 4

Assuming we have the data, what do we need to set for a NN model?

Answer 4

• Network architecture (connections etc.)
• Weights and biases

Question 5

What error function do we use for regression?

Answer 5

MSE
[See flashcards]

Question 6

How do we get the best set of weights for regression?

Answer 6

Minimising the error function (MSE) with respect to the weights.

Question 7

Why does the error function for linear regression have an analytical solution?

Answer 7

The MSE is quadratic in the weights.

We can differentiate to find the minimum, but this can’t be solved analytically to find the weights.

Question 8

What is the relationship between the outputs and inputs in a neural network?

Answer 8

In a neural network, the outputs zi are highly nonlinear functions of the inputs xi.

Question 9

How do the error functions for linear regression and neural networks compare?

Answer 9

The linear regression error is a simple, quadratic function with a single minima.

Neural networks have many complex minima. Different sets of weights can maximise these different minima. There is no analytical solution for the best weights in a neural network.

Question 10

As there is no analytical formula for the best weights in a neural network, how do we find them?

Answer 10

Numerically.

Question 11

How do we find the weights?

Answer 11

Begin with a “guess” for the weights, and change them in steps, decreasing the error each time.

Question 12

When do we know we have reached a minima of the error function?

Answer 12

When the weights stop decreasing (when we no longer decrease the error).

Question 13

What is the formula for going from step W(T) to W(T+1)

Answer 13

[See flashcard]
With each step, we want to make a small change to the weights.

Question 14

When do we have the biggest decrease in error?

Answer 14

When the change in w points in the same direction as the negative of the differential of the error with respect to the weights - the dot product is 1, making it as big as possible.

Change in w proportional to -dE(w)/dw

Question 15

What is the parameter v?

Answer 15

The learning rate.

This controls how quickly we decrease the error.

We have to tune the learning rate to control the optimisation.

Question 16

Describe how v affects the model.

Answer 16

If v is too fast and may overshoot the minimum.

If v is too slow, we might never get there.

Question 17

If we are close to a local minimum, what happens?

Answer 17

We move towards this local minimum, rather than the global minimum.

Question 18

How do we find the global minimum?

Answer 18

1 - train several networks with random weights. We will find multiple minima which we can compare.
2 - use a minimisation method that may let us escape local minima (eg by “hopping over”) local minima
3 - using intuition to make an informed decision

Question 19

Why may it be useful not to use the entire data set to change the weights (batch learning)?

Answer 19

If there is redundancy in the data (repetitions), we are doing more work than we need to.

Question 20

How could we update the weights instead of batch learning?

Answer 20

We could randomly choose a training point i.

Each step now minimises the error for one training point, rather than the total error.

We choose points at random (a stochastic method) - we can get something unexpected from randomness which can allow us to get away from local minima.

The error for a single point decreases, but the total error may sometimes increase which allows us escape from local minima.

Question 21

Rather than batch learning or sequential learning, what do we use in practice?
Explain this.

Answer 21

Multi-batch learning - this is somewhere between batch and sequential learning. Has the benefits of both (sequential learning - stochastic/randomness of training points, batch learning - using more information).

Each time weights are updated, choose a subset of the training set (mini-batch) randomly. We then repeat this with different batches.

This has the benefit that we only need to calculate the derivatives of the error with respect to the weights, and don’t need to solve for w. The derivates are costly to calculate- we use back-propagation to avoid this cost.

Question 22

Why can we never be sure we have reached a global minima?

Answer 22

In theory there are infinite numbers of minima.

Question 23

What is back propagation?

Answer 23

Backpropagation is a powerful algorithm in deep learning, primarily used to train artificial neural networks, particularly feed-forward networks. It works iteratively, minimising the cost function by adjusting weights and biases.

“Cost function” - measures the difference between the network’s predictions and the actual target values. A measure of how well the network is performing.

Question 24

What does inputting xi into the network and getting the output zi involve?

Answer 24

The forward propagation of information through the network.

This is how we make predictions with a. neural network.

Question 25

What does back propagation allow for?

Answer 25

The derivatives of the error to be found by working backwards through the network - once we know activations and node outputs. Derivatives dE/dw6 require many applications of the chain rule - this will be very computationally expensive to find.

Question 26

Briefly, what is the algorithm for updating weights with back propagation?

Answer 26

- Take a set of inputs and find the node outputs (y) and network outputs (forward propagation) - Calculate the error function E(w) - Calculate the derivatives (back propagation - to get deltas)

Question 27

What do we need for backward propagation?

Answer 27

To get the delta of the final output node.

Question 28

What do we use to get the derivatives of errors with respect to weights?

Answer 28

Deltas and outputs of nodes (y)

Question 29

What are the two equations associated with back propagation?

Answer 29

[See flashcard] These formulas arise because of the chain rule.

Question 30

What is a recursion relation?

Answer 30

Recursion relation is a mathematical expression that defines a sequence in terms of its previous terms.

Question 31

What is the principal of back propagation?

Answer 31

Once you have a delta further down the network, you can get deltas back on previous nodes.

Question 32

Even without carrying out the full back propagation or difficult differentiation, what do we know by finding the delta of the final layer?

Answer 32

We know how we should update our final layer weights - using residuals and outputs of the last hidden nodes (see flashcard)

Question 33

How do we find how a weight within a hidden layer appears in the output?

Answer 33

Chain rule

Question 34

In our case, why is the final delta easy to calculate?

Answer 34

We have a linear activation on our final activation

Question 35

What is the delta for the final layer given by (with linear activation function)?

Answer 35

The sum of the residuals. This is mostly the reason why we hose the sum of squares error function to go with this activation function - it allows for a simple expression for the delta.

Question 36

When we have a sigmoid function on the hidden layers, what is delta given by?

Answer 36

[See flashcard]

Question 37

What is the vanishing gradient problem?

Answer 37

The vanishing gradient problem is a challenge that emerges during backpropagation when the derivatives or slopes of the activation functions become progressively smaller as we move backward through the layers of a neural network. This phenomenon is particularly prominent in deep networks with many layers, hindering the effective training of the model. For very large positive or negative activations, the derivative (gradient) tends to zero. This means when you propagate backwards, delta for a given node involves multiplication by 0. Therefore the weights will not update.

Question 38

When do we experience the vanishing gradient problem?

Answer 38

When using the sigmoidal activation function - it is bound by 0 and 1.

Question 39

What function does not cause gradients to vanish?

Answer 39

The ReLU() function - rectified linear unit activation function. This activation function does not saturate in both directions. Sigmoid functions saturate for both big and small activations. ReLU only saturates in one direction and therefore is less prone to the vanishing gradients problem.

Question 40

Why does the vanishing gradient problem become apparent during back propagation?

Answer 40

Because of the recursion relation

Question 41

Describe the ReLU() function.

Answer 41

[See flashcards] - graphs etc. Non-linearity comes about when you have very small activations.

Question 42

What are advantages of the ReLU?

Answer 42

- Can get around the vanishing gradient problem as it only saturates in one direction - Quicker to compute than sigmoid (sigmoid has an exponential which is costly to evaluate)

Question 43

What is the "dying ReLU" problem?

Answer 43

The dying ReLU problem occurs when ReLU neurons in a neural network become inactive and only output zero. This happens when neurons receive negative input, which can cause the network to stop learning. Occurs if too many activations are negative.

Question 44

How can you solve the "dying ReLU" problem?

Answer 44

By using the leaky ReLU function. This looks similar to the ReLU, where positive activations = m. However, there is a small gradient for negative values (rather than a perfectly horizontal line). This means it does not saturate ie the output and derivatives never go to zero.

Question 45

Why would we not use the sum-of-squares error function (MSE) for neural networks?

Answer 45

We don't have to - we cannot find an analytical solution for neural networks, so there is no need to use this error function. The error function can be considered a hyper parameter of the model, just as much as the architecture and choosing activations.

Question 46

Why would we not choose the sum-of-squares error as the perceptron error function?

Answer 46

The sum of squares is not continuous or differentiable for the perceptron, because of the step activation function. With a single node, the error function depends on weights in a discontinuous way.

Question 47

What error function is good for regression problems?

Answer 47

The sum-of-squares error

Question 48

What error function is good for classification problems?

Answer 48

The cross-entropy error function. [See flashcard]

Question 49

When is the cross-entropy error zero?

Answer 49

When all training data points are correctly classified.

Question 50

When is the cross-entropy error infinite?

Answer 50

When all training data points are incorrectly classified.

Question 51

If we use the cross-entropy error function with a sigmoidal activation function, what delta for the final node do we get?

Answer 51

Zi - ti

Question 52

What is one reason to use the sigmoidal activation function over the ReLU activation function?

Answer 52

The sigmoidal activation function pairs well with the cross-entropy error function to give a simple expression of delta for the final node, which we can then use to propagate backwards.

Question 53

What is an adaptive method?

Answer 53

In machine learning, an "adaptive method" refers to an algorithm that can automatically adjust its parameters, like the learning rate, based on the data it encounters during training, allowing it to adapt and improve its performance as it receives new information. We can modify v at each step.

Question 54

How would we modify v if the error increases?

Answer 54

Decrease v - we likely overshot a minima.

Question 55

How would we modify v if the error barely changes?

Answer 55

V may be too small, need to increase it.

Question 56

What is overfitting?

Answer 56

Overfitting is a machine learning problem that occurs when a model is too closely trained to a specific set of data, making it unable to make accurate predictions for new data. The more parameters (weights and biases) we have, the more closely the network can fit the data. but this can come at the cost of not being able to make good predictions for new data. The network is too flexible. We do not want to fit the training data perfectly at the expense of generalisation.

Question 57

What is a flexible model?

Answer 57

One with lots and lots of parameters. A model with the capability to adapt to complex patterns in data by having a high capacity to learn, achieved through a high number of parameters. Networks that are too flexible give extremely accurate answers for training data, but poor answers for new data.

Question 58

What can our intuition tell us about the function to use to fit data?

Answer 58

In real-world problems, functions are usually quite smooth.

Question 59

If we fit the data perfectly with a function, we can get quite high ordered polynomials with associated weights. What can we add to improve this?

Answer 59

Add some extra information (telling us which weights we want to keep or not) to help regularise the model.

Question 60

What is regularisation?

Answer 60

Regularisation in machine learning is a technique that prevents overfitting by reducing the complexity of a model. It adds in extra information about the problem, penalising solutions that are overfitted.

Question 61

For linear regression, what term is added to the error function?

Answer 61

o2 * |w|2 o - sigma

Question 62

What does adding the term o2 * |w|2 to the error function achieve?

Answer 62

This extra term in the loss function gets big if the weights are very big. It avoids oscillatory solutions where the weights are alternating large positive and negative numbers. Ie it makes the solution smoother. o2 indicates how important information is. If o2 is 0, we don't care about weights being large. If o2 is big, the only thing that matters is that the weights shouldn't be too large.

Question 63

What are the benefits of regularisation?

Answer 63

- You can still analytically get the best weights for the error function by minimising it (differentiating and set to 0) - We derived the equation (adding o2 * |w|2) using intuition, but you can also get the same thing using Bayesian statistics

Question 64

What are three ways to avoid overfitting?

Answer 64

1 - early stopping (easy) 2 - dropout (more difficult but more elegant) 3 - Tikhonov or weight decay regularisation (almost no one uses)

Question 65

Describe the early stopping method.

Answer 65

When we look at the learning curves of training and test data, we may see that the error of the training curve continues to decrease over time. However, we may see that the test curve begins to increase at a certain point, which indicates overfitting. At this point, where it begins to increase, we could stop training the model, and use it as is before you overfit it. This is an easy, accessible method as you always have access to the learning cures.

Question 66

When you update weights, the error decreases on the training set. Why is this guaranteed?

Answer 66

Gradient descent.

Question 67

What can be said if the error on the testing/validation set starts to increase?

Answer 67

We are overfitting.

Question 68

Describe the dropout method.

Answer 68

One way to prevent overfitting is to train an ensemble of networks, with different architectures, show them all the same training sets to produce a prediction and average over their results. Each one overfits differently (some large, some small) and so the overfitting is hopefully washed out by averaging. However, training an ensemble of networks requires more computational effort. eg 100x more costly if they all have similar architecture / number of nodes and weights. The dropout method simulates an ensemble of neural networks, without actually having to have an ensemble of neural networks. Each is based on a starting network, this will be the network we want o make predictions on. We train the model over and over, each time updating the weights and dropping out some of the nodes/inputs. We have probability p for dropping out a unit (1 - p = the keep probability). Each time we update the weights, nodes / inputs are dropped by a dropout probability of p. When a node is dropped, the respective connections are also dropped (these weights will not be updated). All in all, this means that for each run, we are updating the weights on a slightly different network. In this way, we are making a single network act like a different network each time. We are training most, but not all of the nodes each time.

Question 69

Following training using the drop out method, how do we make predictions?

Answer 69

We want to use the original network to make predictions. At prediction time, the hidden units receive more activation than during training time - this is because when training the network not all the nodes (and therefore their connections) were present each time. The nodes therefore had a slightly lower activation during training than they will when using the full networks. If p is the probability that a weight was not updated during training, we multiply the activation by (1-p) when making predictions.

Question 70

Describe Tikhonov / weight decay regularisation.

Answer 70

Applying a regularisation (similar to o2 * |w|2 for regression) - ie adding a term to the error function to regularise it.

Question 71

There are an infinite number of possible neural networks. How can we begin to consider choosing the best architecture for our network?

Answer 71

- Symmetries - Time correlations - Dimensionality reductions etc. For our simple feed-forward fully connected network, we may not know in advance what the best network is for the problem of interest and we would need to test different architectures. But as possibilities are endless, we cannot try them all. - Deeper networks are better

Question 72

Which are better, deeper networks or shallower networks?

Answer 72

Deeper networks are better - better to increase the number of layers rather than the number of nodes in a single layer. This is shown to decrease the error rate.

Question 73

Why are deeper networks better if we said that a single hidden layer is enough to give a universal approximater?

Answer 73

This is only the case if there are enough nodes. Generally, deeper networks need fewer units and less data. The further into the network information is propagated, the more complex combinations of features we have, and the more fine detail ie it is modular. The first layer of a NN takes into account the coarse detail, it combines features together and the second layer combines combinations of features etc. We get a more rich and complex function out due to this modularity.