Using a Holdout Set and LASSO

Question 1

Define a houldout set.

Answer 1

A selection of the original data that is not used for estimation.

Question 2

Why do we use a holdout set?

Answer 2

To help reinforce external validity for live data instead of finding the best model for the original data.

Question 3

What are the steps to evalutating a prediction using a holdout set?

Answer 3

1. Split the original data into a larger work set and a smaller holdout set
2. Further split the work set into training sets for k-fold cross-validation
3. Build models and select the best model using k-fold cross-validation
4. Re-estimate the best model using all observations in the work set
5. Take the estimated best model and apply it to the holdout set
6. Evaluate the prediction using the holdout set

Question 4

T/F: The main reason for model selection problem is that we can try out every potentail combination.

Answer 4

False: We cannot try out every single combination, there would be too much!

Question 5

What are the two methods to build models?

Answer 5

1. By hand - specifying variables and model
2. Using smart algorithms

Question 6

What are the pros and cons of using LASSO?

Answer 6

Pros: no need to use ouside info for the model
Cons: may be sensitive to overfitting, hard to interpret

Question 7

What is the LASSO method?

Answer 7

LASSO - least absolute shrinkage and selection operator
- A method to select variables to include in a linear regression to produce good predictions and avoid overfitting

Question 8

What are the two accomplishments that LASSO does at the same time?

Answer 8

1. It selects a subset of the right-hand-side variables, dropping the other variables.
2. It shrinks coefficients for some variables that it keeps in the regression

Question 9

Define the Tuning Parameter.

Answer 9

the weight for the penalty term vs OLS fit. This helps strengthen variable selection.

Note: A lambda of 0 means the regression is OLS.

Question 10

What happens when we have an aggressive threshold? A lenient one?

Answer 10

Agressive: higher threshold and fewer variables are left in the regression
Lenient: lower threshold, and more variables left in the regression

Question 11

T/F: LASSO modifies the way regression coefficients are estimated by adding a penalty term for too many coefficients

Answer 11

True

Question 12

T/F: Big data leads to larger estimation error

Answer 12

False: it makes a smaller estimation error

Question 13

T/F: Lasso creates biased estimates.

Answer 13

True. This is because it shrinks the coefficients, which creates slight bias in the estimates.

Question 14

Why is it okay for LASSO to create bias in its estimates? Does this bias mean LASSO is inferior to OLS?

Answer 14

Remember that lambda searches for the smallest total loss, so it optimizes both bias and variance. Although OLS does not create biased estiments, its variance may increase the total loss. So, LASSO is not inferior to OLS because of this bias.

Question 15

What steps of model building does LASSO substitute? What do we decide?

Answer 15

We decide the initial right hand variables.

LASSO decides the lambda and which variables to include in the final regression.