Model Complexity and Trade Offs Flashcards
(21 cards)
Wow is model complexity increased?
By increasing the highest polynomial power of the model
What is the main characteristic of underfitting?
Training and cross validation error are both high.
What are the main characteristics of underfitting?
Training error is low but cross validation error is high.
What is Bias?
Predictions are consistent but poor model choices lead to wrong predictions. The model does not capture the relationship between the features and the outcomes.
What is Variance?
The model identifies the relationship between features and outcomes but incorporates random noise beside the underlying function.
What are the three sources of model error?
Bias, Variance, and Irreducibility
What is irreducible error?
The error generated form real world data always containing some randomness.
What are the causes of model bias?
The model is misrepresenting the data given missing information, or the model is overly simple. Associated with Underfitting.
What are the causes of model variance?
The output being highly sensitive to changes in input data. This is often due to overly complex or poorly fitted models.
What is the bias-variance trade off?
Model adjustments that decrease bias often increase variance and so finding the best model means choosing the right level of complexity to minimise bias-variance.
What is Linear Model Regularisation?
A method of shrinkage that adds an adjustable regularisation strength parameter directly into the cost function.
Why is Linear Model Regularisation used?
It allows us to manage the complexity tradeoff with more regularisation producing a simpler more biased model.
What are two approaches to regularisation?
Ridge Regression and LASSO
How does Ridge Regression work?
The penalty (lambda) is applied proportionally to squared coefficient values.
How does LASSO work?
The penalty (lambda) is applied proportionally to absolute coefficient values.
What does LASSO stand for?
Least Absolute Shrinkage and Selection Operator
What is L1 Regularisation?
LASSO
What is L2 Regularisation?
Ridge Regression
How is the L1 Norm Calculated?
The sum of the absolute vector values
How is the L2 norm calculated?
The square root of the sum of squared vector values
How does regularisation perform feature selection?
It preforms feature selection by shrinking the contribution of the features.
