Session 10 - Elastic Net

Question 1

What is the strength of penalty determined by for regularised regression?

Answer 1

Strength of tuning parameter lambda

Question 2

Lasso is function of….

Answer 2

Sum of absolute values of coefficients

Question 3

For regularised regression the estimates of β are obtained by what?

Answer 3

Minimising the penalized RSS (𝑅𝑆𝑆(𝜆) )

Question 4

What does not force coefficients to 0 which means it cannot be used for variable selection and is not easy to interpret?

Answer 4

Ridge regression

Question 5

Lasso regression….

Answer 5

Produces 0 coefficients and thus performs variable selection - Adds interpretability

Grouped variables: From several strongly correlated covariates typically one only is selected.

If the number of variables (p) is larger than the sample size (n), the lasso selects at most n variables – may be disadvantage if have very high dimensional data

Question 6

Shrinking coefficient values has the effect of…..

Answer 6

Stabilizing the variability of the estimation, i.e. the complexity of the model; the penalization does add some bias.

Question 7

If we want to predict GP visits based on 8 predictor variables what regularised regression method should we choose?

Answer 7

Ridge

Ridge: allows correlated variables to be included in the model but does not perform variable selection:

• not useful for a large number of variables

Question 8

We have a cohort database with several hundreds of variables collected at baseline of patients “at risk of mental state” (ARMS).

About 15% of the patients developed a psychosis.

Based on the available variables we want to build a prognostic model to predict the likelihood of developing a psychoses.

This model should be used in clinical practice for a risk assessment.

What regularised regression method should we choose?

Answer 8

Lasso – as this is a clinical assessment want to ensure clinicians do not have to take into account too many variables

Lasso: performs variable selection but has got problems with selecting groups of correlated variables, such as set of genes or brain voxels

Question 9

The aim of DNA micro array experiments is to detect differential gene expression.

E.g. to identify genes expression changes under different treatment conditions or among different types of cell samples.

Often thousands or hundred of thousands of genes are tested on the array for expression changes.

Research question:
We want to identify predictors of depression using a case-control dataset with 521135 gene expressions.

What regularised regression method should we choose?

Answer 9

We need a method which allows variable selection but selects a set of correlated variables

A combination of the Ridge and Lasso would sometimes be useful
- Elastic net regression is a hybrid approach that blends both penalization of the L2andL1 norms.

Question 10

We want to predict autism
based on brain activity
differences when someone
looks at the person

3-Dimensional Data
64x64 voxel matrix
43 slices in the brain
= ≈176128 voxels
About 1/3 of this area is studied
- ≈ 50,000 voxels/ hypotheses tests
Add time as 4th dimension
and will will get into 100.000s
-Massive multiple
comparisons problem!

What regularised regression method should we use?

Answer 10

We need a method which allows variable selection but selects a set of correlated variables

A combination of the Ridge and Lasso would sometimes be useful
- Elastic net regression is a hybrid approach that blends both penalization of the L2andL1 norms.

Question 11

What is elastic net regression?

Answer 11

Hybrid approach that blends both penalization of the L2andL1 norms.

It is a generalization of ridge, lasso and unregularized linear regression

Question 12

What is the elastic net regression formula?

Answer 12

The l1 (“lasso”) part of the penalty
generates a sparse model and performs variable selection

The l2 (“ridge”) part of the penalty
Removes the limitation of the number of selected variables
Encourages grouping effect: Selects groups of correlated variables

Question 13

What do we get if we set λ1 to 0 for elastic net?

Answer 13

l2 penalty or Ridge regression

Question 14

What do we get if we set λ2 to 0 for elastic net?

Answer 14

L1 penalty or lasso regression

Question 15

What do we get if we set λ1 and λ2 to 0 for elastic net?

Answer 15

“Normal” linear regression

Question 16

The usual approach to optimizing the lambda “hyper-parameter” is through cross-validation by minimizing the mean squared prediction error.

But in elastic net regression…

Answer 16

the optimal lambda1 hyper-parameter (tuning parameter) also depends on the lambda2 hyper-parameter (“hyper-hyper-parameter”).

Question 17

How do we find best lambda for elastic net?

Answer 17

Tuning of Elastic Net

Question 18

In the elastic net we need to tune two parameters

How can this be pursued?

Answer 18

1. We start with a fixed λ2 and then select the best λ1 using cross-validation and record the cross-validated prediction MSE
2. We then choose another λ2 and select the best λ1 and record the cross-validated MSE.
3. We repeat this procedure for a large number of λ2 and select then the model with λ1 and λ2, which minimizes the cross-validated MSE

However, glmnet uses a slightly different approach because this selection of lamda causes computational problems and may not always find best solution

Question 19

Why does EN not use L1 and L2 penalties to find estimate of regression coefficient?

Answer 19

Because finding two lambdas using cross validation doesn’t work well.

Question 20

L1 and L2 Elastic net estimate of β is obtained by minimising what?

Answer 20

𝑅𝑆𝑆(𝛽(𝐿1&2))=..

Question 21

Finding the two lambdas using CV does not work very well in EN.

How can prediction performance be improved?

Answer 21

Mixing parameter alpha is used:

Here have only one type of lambda and not lambda 1 and 2 penalty terms

Modify alpha for L1 penalty term with lambda and 1- alpha for L2 penalty terms

Unless have lambda of .5 the lambdas will differ between two penalty terms and by changing range of alpha can test a range of different penalties

Alpha range between 0 and 1 whilst lambda can be greater than 0

Question 22

What is lambda?

Answer 22

The shrinkage parameter

Question 23

What is alpha?

Answer 23

The elastic net is the ratio between L1 and L2 penalty! (for details: See machine learning course)

Question 24

If α= 1 =

Answer 24

Lasso

Question 25

If α = 0 =

Answer 25

Ridge

Question 26

En requires an α value between what?

Answer 26

0 and 1

Question 27

The best α and lambda are those values that ..

Answer 27

Minimizes the cross-validation error.

Question 28

How can we tune the model?

Answer 28

Loop 1: (outer loop)
For alpha = 0, 0.01, 0.02, ….1

Loop 2 (inner loop)
For lambda = 0, 0.01, 0.02, ….
Find best lambda through cross-validation:
= Lambda with smallest MSE of prediction

Save lambda and MSE
Next lambda
Next alpha

Compare the MSE for all alphas.

The alpha with smallest MSE is the best tuning parameters

Use this alpha and respective lambda for final model with all data

Question 29

The EN performs simultaneous…

Answer 29

Regularization and variable selection.

Question 30

EN has ability to perform what?

Answer 30

Grouped selection

Question 31

What is EN appropriate for?

Answer 31

The p&raquo_space; n problems – more variables than sample size

Appropriate for bioinformatics and brain imaging studies

Question 32

For EN two tuning parameter need to be…

Answer 32

Trained by performing a grid search and identifying the pair of parameters, which minimize the cross-validated MSE of prediction

Question 33

EN is extend to…

Answer 33

Generalized linear model (using glmnet) but also to sparse PCA, DFA, support vector machines, etc.