L5 - Regression - Bias-Variance Trade Off

Question 1

Define bias…

Answer 1

Bias refers to the extent that a model produces errors due to under-fitting. High bias means that high model error is encountered due to underfitting.

Question 2

Define variance…

Answer 2

Variance refers to the extent that a model has been overfit. High variance means a model will not perform accurately on unseen data due to overfitting the training data.

Question 3

What is the ideal bias-variance trade off for a regression model?

Answer 3

Low bias and low variance.

Question 4

What are the 3 causes of model error? Define each…

Answer 4

Bias - The extent to which a model predicts wrong.

Variance - The extent to which a model has learned data, and thus is overfit.

Irreducible Error - Refers to the unavoidable randomness of real world data that can cause errors.

Question 5

How can we tune bias and variance?

Answer 5

To increase bias we must reduce variance and vice versa.

Question 6

What is another way of wording the bias-variance trade off of a model?

Answer 6

Bias-variance trade off can also be called a Complexity trade off.

Question 7

If we achieve the optimal bias-variance ratio, what model complexity does this give us?

Answer 7

Optimal model complexity.

Question 8

What technique can we use to prevent our model from overfitting, and to encourage generalisation?

Answer 8

Linear Model Regularisation (Shrinkage)

This reduces the magnitude of the coefficients in the polynomial regression model.

Question 9

What is the objective of Linear Model Regularisation?

Answer 9

Establish a trade off between bias and variance, resulting in optimal model complexity.

Question 10

How does Linear Model Regularisation work?

Answer 10

• Introduces a penalty to the models loss function
• Penalty can be increased or decreased to increase or decrease complexity.
• We want to push all coefficients towards 0.

Question 11

What happens if the Tuning parameter of the Regression Models cost function is increased?

Answer 11

• Results in less regularisation, which leans to overfitting ( increases variance )

Question 12

What happens if the Tuning parameter of the Regression Models cost function is decreased?

Answer 12

• Results in more regularisation
• Decreased model complexity which leans to underfitting ( increases bias )

Question 13

What is L1 regularisation?

Answer 13

• L1 ( LASSO ) applies a penalty value that is proportional to the sum of the absolute coefficient values.
• Prevents overfitting and performs feature selection.

Question 14

What is L2 regularisation?

Answer 14

• Ridge Regression
• Prevents overfitting and improves model stability.
• A penalty is applied that is proportional to the squared coefficient values.
• Penalty imposes a bias, and thus can be used to control the bias-variance trade off.

Question 15

How do we establish the penalty value?

Answer 15

• Through cross-validation, to find the lambda with the lowest variance.

Question 16

What do L1 and L2 have in common?

Answer 16

Both push coefficients towards 0.

Question 17

Why do L1 and L2 push coefficients to 0?

Answer 17

• Helps prevent overfitting
• Improves generalisation
• Reduces variance.

Question 18

How does L1 Regularisation perform feature selection?

Answer 18

By pushing coefficients to 0, the features with non-zero coefficients are known to be important.