Linear regression

Question 1

What are the assumptions of linear regression

Answer 1

**Linearity ** (residuals vs fitted plot)
Independence (Residuals (differences between observed and fitted values) should appear randomly scattered with no patterns when plotted. (residuals vs fitted plot)
Constant variance (homoscedsticity) - similar variance of observed vs fitted values.
Normality - regression residuals (prediction errors) are well described by normal distribution (Q-Q plot)

Question 2

What are residuals?

Answer 2

The difference between the actual and predicted values. Sometimes referred to as “errors”

larger residual, the worse the model is at predicting the outcome.

Question 3

What is homoscedasticity?

Answer 3

Variability. In linear regrression we assume constant variance. Regression residuals should show similar variance across all levels of fitted values.

Question 4

What do we mean by fitted values?

Answer 4

Values predicted by the regression model.

Question 5

What do we mean by independence of the model?

Answer 5

Residuals (errors) are independent of each other. Knowing the error of one data point should tell you nothing about the error of another.

Question 6

How do we check for linearity in a regression plot?

Answer 6

Look at the residual vs fitted values plot. If the line is mosty horizontal, we can assume linearity.

by default, plot describes the 3 largest absulte value residuals -
plot 1

Question 7

How do we check for homescedasticity (constant variance) in a model using residual vs fitted plot?

Answer 7

• Looking for the residuals to be equally distributed from the line as fitted values increase
• Problems if there is a “fan” shape (residuals increase or decrease as fitted increases)
• Increasing residuals as fitted increase

Plot 1 (residual vs fitted) or Plot 3 (scale-location)

Question 8

How do we check for independence in a model?

Answer 8

Looking for random scatter of values
Problems if there is a pattern, trend, or clump of values
If data are measured over time, check especially for evidence of patterns that might suggest they are not independent (eg, plot residuals against time to look for patterns)

Plot 1

Question 9

How do we check for normality in a model?

Answer 9

• Assess with Normal Q-Q plot (and possibly histogram) of the residuals
• Settle for the residuals satisfying the “nearly Normal” condition
• Problems if standardized residual values high (generally >2 or < -2)

Normality assumption becomes less important as the sample size grows

Plot 2

Question 10

How do you assess for collinearity of predictors?

Answer 10

• Assess with scatterplot matrix and correlation coefficient of predictors
• Also, after fitting model, look at variance inflation factors (VIF); values >5 suggest a concern
• If collinearity a concern, then drop a predictor

Question 11

What is MAE?

Answer 11

mean absolute prediction error, lower is better

Question 12

What is RMSE?

Answer 12

root mean squared prediction error, lower is better

Question 13

What do AIC and BIC tell you in a model?

Answer 13

indexes where lower value = better model

Akaike Information Criterion and Bayesian Information Criterion

Question 14

What is Adjusted R2?

Answer 14

adjusted down for the number of covariates in the model; not a variation of anything but an index where higher is better

Question 15

What is Nominal R2?

Answer 15

Proportion of variation in the outcome explained by the model, higher is better

Question 16

Key things to determine which model is best?

Answer 16

Nominal R2 = higher is better
Adjusted R2 = higher is better
AIC and BIC = lower is better
RMSE = lower is better
MAE = lower is better

Question 17

Residuals in a model that meet all assumptions

Answer 17

standardized residuals (in addition to following a Normal distribution in general) will also have mean zero and standard deviation 1, with approximately 95% of values between -2 and +2, and approximately 99.74% of values between -3 and +3

residual 113 and 72 are beyond these parameters, so we need to look at leverage/influence

Question 18

How do we assess homoscedasticity using the scale-location plot?

Answer 18

Plots the suqare rool of the standardized residuals agains fitted values
Looking for loess smooth to be flat; problem if a slope.

Question 19

How do we assess leverage?

Answer 19

A residuals vs Leverage plot or Cook’s distance plot
most highly leveraged points are shown furthest to the right in this plot (unusual predictor values)
Points with high influence are shown in the upper or lower right section outside of the dotted line (cook’s distance > .5

Plot 5

Question 20

What can we do if assumption of linearity is not met in a linear regression model?

Answer 20

o Look for potential transformation of outcome Y with Box-Cox plot
o Look at residual plots against each of the individual predictors
o This will help identify the specific predictor (or predictors) where we need to use a transformation
o If all predictors are problematic, we might transform the outcome Y
o If plot of x vs. y suggests non-linearity (curves) then consider a polynomial or spline for the predictor

Question 21

What can we do if assumption of homoscedasticity is not met in a linear regression model?

Answer 21

o Consider potential transformations of the predictors, possibly fitting some sort of polynomial term or cubic spline term in the predictors

Question 22

What can we do if assumption of independence is not met in a linear regression model?

Answer 22

o No specific solution outlined in the course notes or slides; probably just use a different model

Question 23

What can we do if assumption of normality is not met in a linear regression model?

Answer 23

Consider a transformation of the outcome Y
(use Box-Cox; ladder of power lambda = 1 = unchanged)

Question 24

What transformation for may be useful for right-skewed data?

Answer 24

consider power transformations below 1 - sqrt = 0.5, natural log = 0, inverse = -1

Question 25

What transformation for may be useful for left-skewed data?

Answer 25

consider power transformations above 1 - square = 2, cube = 3

Question 26

What transformation for may be useful for data with heavy (outlier-proned) or light tails (fewer extreme values than normal distribution)?

Answer 26

None. If a value has high influence (very different from rest of values) it may merit removal before modeling with explanation in our report. (i.e. This observation was included in the sample in error, for instance, because the subject was not eligible; An error was made in transcribing the value of this observation to the final data set; This observation is part of a meaningful subgroup of patients that I had always intended to model separately) | ‘I ran my model and it doesn’t fit this point well’ is NOT a good reason

Question 27

What does skew in a normal Q-Q plot look like?

Answer 27

Question 28

Need to include review of interpretation of a linear regression

Question 29

You are building a linear regression model for a quantitative outcome called out with only a limited number of observations, and need to include four predictors: age (in years), prior (1 = had prior surgery, 0 = no prior surgery), severity (three categories: High, Medium, Low) and length (in centimeters). Suppose you are permitted to spend an additional four degrees of freedom beyond the five accounted for by the intercept term and the main effects of these four predictors. Based on the Q14 Spearman 𝜌2 Plot below, which of these models best does this additional spending?

Answer 29

i. The Spearman 𝜌2 plot pushes us to age and then severity as candidates. ii. age is a quantitative variable. iii. severity is a categorical variable with three categories. iv. To spend an additional four degrees of freedom, we can use a 4-knot spline in age (which adds 2 df) and an interaction of the linear age effect and severity (which also adds 2 df), as in option d.

Question 30

Consider the information below on the distribution of a potential outcome variable in a linear regression model to be built. Based on the three pieces of output provided, which transformation of the outcome data would be most appropriate?

Answer 30

A logarithmic transformation is likely to be helpful. The Box-Cox plot likes power 𝜆 = 0, indicating a logarithm. There is right skew in the boxplot, and all of the values are strictly positive, with the mean well above the median.

Question 31

* Suppose you want to include five predictor variables (dm, group, symptom, sys_bp, and tot_chol) in a linear model for result. dm and group are categorical variables with 2 and 4 categories, respectively. symptom, sys_bp, and tot_chol are continuous variables. * The main effects model including these five predictors uses 7 degrees of freedom. Suppose you are now willing to spend 12 df total (an additional 5 df in an augmented model). Use the Spearman rho-square plot to motivate your choices.

Answer 31

* Our augmented model should add 5 df to the main effects model, and the most logical choices are to add nonlinear terms to tot_chol and group according to the plot. * Therefore, we will add a 4-knot spline for tot_chol for 2 additional df (the spline uses K-1 = 3 df, or 2 more df than the main effects model). We will also add an interaction between group and the main effects of tot_chol for an additional 3 df (df1 × df2 = 3 × 1 = 3 more df).