Chapter 6 The Bias-Variance Trade-Off

Question 1

HOW CAN WE BREAK DOWN THE PREDICTION ERROR FOR ML ALGORITHMS? P28

Answer 1

Bias error, Variance error, Irreducible Error

The irreducible error is the error that we can not remove with our model, or with any model. The error is caused by elements outside our control, such as statistical noise in the observations. … usually called “irreducible noise” and cannot be eliminated by modeling.

Question 2

WHAT DO LOW BIAS AND HIGH BIAS MEAN? GIVE SOME ML ALGORITHM EXAMPLES FOR EACH P29

Answer 2

ˆ Low Bias: Suggests less assumptions about the form of the target function. EXP: Decision Trees, K-Nearest Neighbors, SVM
ˆ High-Bias: Suggests more assumptions about the form of the target function. EXP: Linear Regression, Logistic Regression, Linear Discriminant Analysis

Question 3

WHAT DO LOW VARIANCE AND HIGH VARIANCE MEAN? GIVE SOME ML ALGORITHM EXAMPLES FOR EACH P29

Answer 3

ˆ Low Variance: Suggests small changes to the estimate of the target function with changes to the training dataset. EXP: Linear Regression, Linear Discriminant Analysis and Logistic Regression
ˆ High Variance: Suggests large changes to the estimate of the target function with changes to the training dataset. EXP: Decision Trees, k-Nearest Neighbors and Support Vector Machines.