Regularisation

Question 1

Regularisation

Answer 1

Try to make sure, that the model does not overfit to the training data - i.e. let’s not don’t trust the training data too much. The more iterations, the more we fit the model to the training data, minimising loss.

Regularisation does not depend on the features. It looks over the model’s weights and tries to keep them overall small.

Question 2

Why regularisation?

Answer 2

We try to generalise. The model does not need to be perfect for the training data - but should be useful for unseen data.

Question 3

A simple analogy for overfitting

Answer 3

If you learn english by speaking only to a teenager. You’ll pick up slang so good, that you might end up not being able to speak english with any other person. Regularisation helps to stop learning - to generally be able to speak english.

Question 4

How to do regularisation

Answer 4

We still minimise the loss, but we add penalty for a complex model. Loss(data|model) + complexity(model)

Question 5

A strategy to model complexity: complexity(model)

Answer 5

One popular strategy is to try to prefer smaller weights, that is make the parameters as small we can get away with while still getting the training examples right.

Question 6

L2 regularisation

Answer 6

also called ridge regularisation
sum of the squared weights

(for linear models)
penalise big weights
weights should be centred around zero
weights should be normally distributed

L(w|D) + lamda*sum(square(w))
(lamda a weight indicating, how much we care about)