GLM 1 - Simple linear regression

Question 1

In a linear relationship, how is the value of an outcome variable Y approximated?

Answer 1

Y ≈ β0 + β1X.

Y= dependent variable
B0= is an intercept
B1 = slope coefficient of X

Question 2

What is the intercept/B0 (often labelled the constant)?

Answer 2

Expected mean value of Y when all X=0.

Question 3

What is the B1?

Answer 3

The slope or how y changes per unit increase in x.

B1 is increase in y when you change x by a unit/when x is increased by one unit y will increase by beta 1

Question 4

What is the terminology of a linear regression?

Answer 4

1. We say that Y is regressed on X .
2. We are expressing Y in terms of X .
3. The dependent variable, Y , depends on X .
4. The independent variable, X , doesn’t depend on anything.

Question 5

How are the coefficients or parameters B0 and B1 estimated?

Answer 5

Using the available data:
(x1, y1), (x2, y2), . . . , (xn, yn ) - We have here a sample size of n data points.

Question 6

How are the estimates of parameters written?

Answer 6

The estimates of the parameters are written with a circumflex or hat: ^

We then write our linear equation with these estimated coefficients: y^ = β^0 + β^1 xi

Only a hat over the dependent variable.

Independent variable (xi) does not have a hat as treated as fixed.

Question 7

B0 and B1 are independent of each other

True or false

Answer 7

True

Question 8

What does the circumflex allow us to differentiate between?

Answer 8

True value and estimated value

Question 9

What happens if add value to B0?

Answer 9

This would only affect y but not B1xi – B0 can change independently of B1

Question 10

What is y^I?

Answer 10

Predictions or predicted values of the outcomes y , given the independent variables, xi ’s

Question 11

What are the differences between the predicted values, y^ i’s, and the observed values, yi ’s?

Answer 11

The residuals:
e^ := yj − yi^ .

That is, these are the values that remain after we have removed the
predictions from the observations.

Question 12

Why are the residuals, e^i ’s, also equipped with a hat?

Answer 12

Because these are also estimated values.

Question 13

Why are the black error bars vertical, and not perpendicular to the line in blue?

Answer 13

Residuals correspond to an addition to value of y hat

Question 14

How can the optimal value of the parameters, β0 and β1 be found?

Answer 14

By considering the sum of the squares of the residuals:

RSS := e^1 + e^2

Question 15

Why do we square residuals?

Answer 15

Residuals are defined as a subtraction of the predicted values from observed values; we can rewrite RSS in the following fashion: RSS = (y − y^1)2. Some values may be negative and some may be positive and thus must square them to normalise them and ensure they make a positive contribute to RSS.

Question 16

What is the optimal purpose for B0 and B1?

Answer 16

To minimise distance from all the data points.

Question 17

What is RSS a function of and why?

Answer 17

B0 and B1 because all residuals do depend upon values of B0 and B1. Thus, we may write the RSS as depending on these quantities:

RSS(β^0, β^1) = e^12 + e^22

Question 18

The value taken by the RSS can therefore minimized for some values of β^0 and β^1.

How do we write this?

Answer 18

(β0, β1 ) := argmin RSS(β0 , β1),

Argim RSS- means argument that minimizes RSS

Where the hats on the right hand-side of the RSS have been suppressed.

Question 19

The RSS is a function of the parameters β0 and β1 therefore…

Answer 19

it can take a range of values across a two dimensional landscape

Question 20

How can we assess the accuracy/goodness of fit of our model?

Answer 20

Using the previously minimized value of the RSS

Question 21

What is one way of quantifying accuracy of model?

Answer 21

Compare RSS with total sum of squares (which can be reformulated as the sum of squares of null model as null model is model with only y intercept)

Question 22

What is R2 also known as?

Answer 22

Coefficient of determination

Question 23

What does R2 measure?

Answer 23

Proportion of variance in the dependent variable explained by the independent variable.

Question 24

For simple regression, the R2 can be shown to be equivalent to what?

Answer 24

Correlation of the IV with the DV.That is,where R2 and the square of Cor(Y , X ) are equal.

Question 25

What is a random variable?

Answer 25

A function, from a sample space Ω to the real numbers, R, such that
X : Ω ›→ R.

Uppercase X is random variable

Question 26

For every point in the sample space, ω ∈ Ω, the random variable X may (or may not) take what?

Answer 26

A different value, such that we have:
X (ω) = x.

We call x , the realization of X(random variable) at ω(point in sample space)

Question 27

What does the probability to obtain x , count?

Answer 27

The number of ω producing x , written as

P[X = x ] := P[{ω ∈ Ω : X (ω) = x }].

Question 28

What is the random variable for the toss of an even coin, with head, H, and tail, T?

Answer 28

X : {H, T } ›→ {0, 1},

with X (H) = 0 and X (T ) = 1, producing the probabilities

In other words, X has H and T as a sample space and X is going to assign to those events 0 for heads and 1 for tails

Question 29

If we have a single-faced coin, Y : {H, T } ›→ {0, 1}, such that Y (H) = 0, and Y (T ) = 1, what are the probabilities?

Answer 29

P[Y = 0] = 1, and P[Y = 1] = 0

The measure P is used to give probability mass to each element in Ω.

Question 30

What is the discrete expectation?

Answer 30

For a discrete value y, the expectation E[Y] is the sum of the value of y’s(the realizations- all the possible values taken by y over values in sample space). Finite number of values a y as this is for a discrete random variable. Weight each values by probability of obtaining those values. This is almost identical to arithmetic mean.

Question 31

What is the Arithmetic mean ?

Answer 31

A special case of expectation in which p of y are uniform across all values of possible values of y(1/n).

Question 32

In simple regression what are we given?

Answer 32

Two sequences of data points.

Each pair of observations is a case, (yi , xi ), with i = 1, . . . , n.

Question 33

What is the deterministic and stochastic part of a statistical model?

Answer 33

yi = β0 + β1xi + εi

B0 + B1xi = deterministic

ei = stochastic

Question 34

What is one difference between regression and correlation?

Answer 34

In regression one of the variables is treated as the outcome variable or dependent variable, generally denoted by the yi ’s

We will then use the other variables for predicting that outcome. As a result, the other variables are referred to as predictors, or independent variables, and are denoted by xi ’s, or features in the machine learning literature.

Question 35

What is the deterministic part of a univariate simple linear regression made up of?

Answer 35

1. Mean expressed as a conditional expectation:

E[Y |X = xi ] = β0 + β1xi ,

2. Variance function, expressed as a conditional variance operator:

Var[Y |X = xi ] = σ2, ∀i = 1, . . . , n.

The (unknown) parameters in this model are therefore (β0, β1, σ2).

Question 36

What are the unknown parameters in the deterministic part of a simple linear regression?

Answer 36

1. β0 is the y -intercept of E[Y |X ], when X = 0. Thus, we have
   E[Y |X = 0] = β0.
2. β1 is the rate of change of E[Y |X ], such that
   E[Y |X = x + 1] − E[Y |X = x ] = β1.
3. σ2 is the (conditional) variance of Y , given X . It is strictly positive,

σ2 > 0.

Question 37

What does the stochastic part of a simple linear regression?

Answer 37

Random noise- In general, the observables or observed data, denoted by yi ’s, differ from the expected values of Y , given xi , such that

yi = E[Y |X = xi ] + εi , i = 1, . . . , n,
where the εi ’s are the statistical errors, collectively referred to as additive noise.

The εi ’s are defined as the difference between the observables and the conditional expected values –that is,
εi = yi − E[Y |X = xi ].

Geometrically, the errors correspond to the vertical distances between each yi and its conditional expectation. Note that the error terms are not observable, since they depend on the unknown parameters (β0, β1).