Concepts week 4 - Generalisation

Question 1

What is Ockham’s razor law?

Answer 1

“More things should not be used than are necessary.”
If there are two explanations of a hypothesis - the simper one might be the right one.

The less complex a ML model, the more likely that a good empirical result is not just due to the peculiarities of the sample.

Razor is a metaphor. Shaving off all unnecessary things.

Question 2

What is a test set methodology?

Answer 2

In contrast to generalisation theory - we use empirical approach. We take one set of data from a distribution and train the model on it - the training set. Then we use another set of data (test set) from the distribution to see how good the model is. If our model does a good job on test data - we have a good feeling that our model generalises well. ==> Generalisation

A machine learning model aims to make good predictions on new, previously unseen data. But if you are building a model from your data set, how would you get the previously unseen data? Well, one way is to divide your data set into two subsets:

training set—a subset to train a model.
test set—a subset to test the model.
Good performance on the test set is a useful indicator of good performance on the new data in general, assuming that:

The test set is large enough.
You don’t cheat by using the same test set over and over.

Question 3

Describe Generalisation.

Answer 3

A ML model’s ability to make correct predictions on new, previously unseen data as opposed to the data used to train the model.

We train our model on a training set and verify the model with our test set. If our model does a good job on test set - we have a good feeling that our model generalises well on new and unseen data ==> Generalisation

Question 4

Describe overfit model

Answer 4

An overfit model gets a low loss during training but does a poor job predicting new data.

Overfitting is caused by making a model more complex than necessary (Ockham’s razor).