GLM 3- Assumptions

Question 1

What is the first assumption of GLM?

Answer 1

Response-predictor Linearity

Question 2

How can the first assumption of GLM be diagnosed?

Answer 2

Residual plots allow to identify non-linearity.

They plot the fitted values, y^i’s, against the residuals,e^i s.

The plot should indicate a linear trend –i.e. a line– between the fitted values and the residuals.

Question 3

How can the first assumption of GLM be remedied?

Answer 3

If the residual plot indicates that there is a non-linear relationship, one can either:

Transform the predictors, log X, square root of X

Use polynomial regression by including X2, X3, for instance

Question 4

What is assumption 2 of GLM?

Answer 4

Constant Variance of Errors

Question 5

How can the second assumption of GLM be diagnosed?

Answer 5

Residual plots can, again, enable us to assess whether the variances of the error terms are constant.

The error terms are assumed to be homoscedastic. That is, to have identical variance for different levels of the fitted values, y^i ’s.

Consequently, we should observe a uniform distribution of the variation of the residuals across the levels of predicted values.

Question 6

How can the second assumption of GLM be remedied?

Answer 6

We can transform the response, Y , by taking log(Y ), or square root of Y. If such transformations do not work or are impossible, just report it in your analysis.

We can exploit the source of variability in the responses, if known. The yi ’s may be aggregates with associated variances, σi . In such cases, we can use weighted least squares (WLS).

Question 7

What is the third assumption of GLM?

Answer 7

Non-correlation of Errors

Question 8

How can the third assumption of GLM be diagnosed?

Answer 8

Serial Residual plots allow to identify the correlation of the errors.

They plot the residuals, e􏰇’s with respect to the observations IDs.

The plot should not indicate long-term dependency between sequences of residuals. This would violate the assumption of independent observations.

Question 9

How can the third assumption of GLM be remedied?

Answer 9

Typically, non-independence may be present due to some structure in your data, due to groups or time.

Model the group structure in your data, using a mixed-effects model, or hierarchical model.

Model the time-lag structure in your data, again using a mixed-effects model, or hierarchical model.

Question 10

What is assumption 4 of GLM?

Answer 10

Detecting Outliers

Question 11

How can assumption 4 of GLM be diagnosed?

Answer 11

An outlier is a point, which is far from the values predicted by the model.

Outliers will result in an increase in the Residual Sum of Squares (RSS),
used to compute R2, and the confidence intervals for each parameter.

The studentized residual plot show the values of the residuals, ei ’s, divided by their standard errors

Question 12

How can assumption 4 be remedied if violated?

Answer 12

Typically, the studentized residuals should not exceed 3 standard errors.

If a data point has a residual with a studentized residual of 3 or more, you may consider removing it, especially if you suspect that this observation is faulty in some ways.

However, care should be taken as the presence of an outlier may also indicate a deficiency in your model.

Question 13

What is assumption 5 of GLM?

Answer 13

High-leverage Points

Question 14

How can assumption 5 of GLM be diagnosed?

Answer 14

A high-leverage point is a data point, whose removal produces a substantially different set of parameters.

The leverage, or hat-value, is a quantity that measures how unusual is that data point with respect to all the others.

We can plot the individual leverages against the values of the studentized residuals to identity points that are outliers and have also high leverage.

Question 15

How can assumption 5 of GLM be remedied?

Answer 15

There is no specific rule of thumb for detecting high leverage.

As for outliers however, you may consider removing it, especially if you suspect that this observation is faulty in some ways.

But, again, care should be taken as the presence of an outlier may also indicate a deficiency in your model.

Question 16

What is assumption 6 of GLM?

Answer 16

Multicollinearity

Question 17

How can assumption 6 of GLM be diagnosed?

Answer 17

Two predictors are said to be collinear is they are strongly correlated with each other.

Pairwise collinearity can be assessed by considering the correlation matrix of your predictors.

Multicollinearity, however, may occur in the absence of severe pairwise collinearity. In that case, we use the variance inflation factor (VIF),

VIF (β^j ) := 1/ 1−R2 Xj |X−j

If the variance in Xj is strongly explained by all the other predictors in the model, then R2 will be close to 1, and therefore its VIF will be very large. Xj |X−j

Question 18

If violated how can assumption 6 of GLM be remedied?

Answer 18

If one variable, say Xj , exhibits a VIF larger than 5, we may resort to one of the following:

1) Drop that variable.

2) Combine the variables that are collinear, by taking the average of the
standardized variables.

3) Create a latent variable that combine the multicollinear variables.

Note that removing a collinear variable will not compromise the overall fit of your model, because most of the information contained in that variable is also contained in other variables.

Question 19

What is the default Plotting for lm Objects in R?

Answer 19

Every lm object can be plotted, using one of the following:

plot(fit)
plot(lm(y ∼ x1 + x2))
plot(fit <- lm(y ∼ x1 + x2))

Alternatively, we may select which plot we need as follows:

plot(fit, which=c(1,4))

Question 20

How do we know the options of the plot function in R?

Answer 20

?plot.lm

Question 21

1. How can residual plot be created in R for assumption 1?
2. What does this do?

Answer 21

1. plot(fit, which=1)
2. Check linearity of errors.

Question 22

1. How can a QQ-plot of Standardized Residuals be fitted in R to test assumption 2?
2. What does this do?

Answer 22

1. plot(fit, which=2)
2. Check normality of residuals.

Question 23

1. How can a Scale-location Plot be fitted in R to test assumption 3?
2. What does this do?

Answer 23

1. plot(fit, which=3)
2. Check homoscedasticity (equal variance)

Question 24

1. How can a Cook’s Distances plot be fitted in R to test assumption 4?
2. What does this do?

Answer 24

1. plot(fit, which=4)
2. Influential (or leverage) cases.

Question 25

1. How can a Residuals vs Leverages plot be fitted in R to test assumption 5?
2. What does this do?

Answer 25

1. plot(fit, which=5)
2. Checks Cases that are influential and outliers

Question 26

1. How can a Cook’s Distances vs Leverages plot be fitted in R to test assumption 6?
2. What does this do?

Answer 26

1. plot(fit, which=6)
2. Check influence and leverage.

Question 27

What is an estimator?

Answer 27

A function of the data, (y, X), and therefore can be denoted as follows:

β^ := β^(y, X).

This holds for every pair (y, X). Since the dependent variables are random, it follows that, if β^ is a function of y, then β^ is also a random quantity. Thus, ignoring X, we may write
β^ := β^(Y1,…,Yn).

Question 28

What are the statistical properties of OLS Estimators?

Answer 28

Unbiasedness. We say that β^ is an unbiased estimator of β, if the following condition holds, for every β ∈ R,

E[β^(Y1,…,Yn)|X] = β.

Minimal Variance. For every other estimator, β~, the OLS estimator, β^, has variance,

Var[β^(Y1,…,Yn)|X] ≤ Var[β~(Y1,…,Yn)|X].

Mean Squared Error (MSE). Finally, we will see that these two criteria can be combined by considering the MSE of β^ with respect to β,

MSE(β, β^) := E[(β − β^)2|X ]

Question 29

What is the MSE?

Answer 29

We assume that β represents a single parameter. Then,

E[(β^ − β)2|X] = E[(β^ − E[β|X] + E[β^|X] − β)2|X] ….

where the bias of β^ is defined as follows b2(β^) := (E[β^|X ] − β)2.

Thus, we obtain the following decomposition,
MSE(β,β^) = Var[β^|X] (variance) + b2(β) (squared bias)

The MSE is a theoretical quantity, since it depends on the unknown true parameter β.

Question 30

What is the Best Linear Unbiased Estimator (BLUE)?

Answer 30

An estimator β^ of a parameter β

i. If it is a linear function of the observed values y;

ii. If it is an unbiased estimator of β;

iii. And if it has minimal variance among all other linear unbiased estimators β~, such that,

Var[β^|X] ≤ Var[β~|X].

Question 31

What do the Gauss-Markov assumptions state?

What does this mean?

Answer 31

Zero-centered errors: E[ε|X] = 0,

Uncorrelated errors and homoscedasticity: Var[ε|X] = σ2in

In, Linearity of mean function: y = f (β, X), is linear in β.

This means the OLS estimator, β^, is BLUE for β.

Question 32

If we want to use MSE as a criterion what must we recall?

Answer 32

The standard decomposition

MSE(β,β^) = Var[β^|X] + b2(β^).

Question 33

What is MSE in statistics?

Answer 33

The mean squared error (MSE) is a theoretical object, dependent on (i) the true parameter, and (ii) the true model of Y .

It is defined as the integrated squared distance between the true parameter, β, and the estimator, β^, conditional on X; such that:

MSE(β, β^) := E[(β − β^)2|X ].

Question 34

What does MSE depend on in machine learning?

Answer 34

The mean squared error (MSE) depends on the joint model of Y and X.

Question 35

How is MSE defined in machine learning?

Answer 35

As the integrated squared distance between the observed outcome, y, and the predicted outcome, f^ (X); such that:

MSE(f^ ) := E[(y − f^ (X))2].

Question 36

What are models are expressed in terms of? Dependent variables are treated as random.
Independent variables are treated as fixed.

Answer 36

Conditional expectations, E[Y |X].

Question 37

How are dependent variables treated as?

Answer 37

Random

Question 38

How are independent variables treated as?

Answer 38

Fixed

Question 39

What is measurement error only assumed in?

Answer 39

Yi ’s.