Least Squares Intro

Question 1

What are the names of the two variables involved in bivariate regression?

Answer 1

On the LHS y is the dependent variable or the regressand or the outcome or the explained variable
On the RHS x is the independent variable or the regressor or the explanatory variable or the covariate or the control

Question 2

What is regression to the mean?

Answer 2

If the first observation of a variable was extreme, the next observation is likely to be less extreme, and vice versa

Question 3

What is the most basic relationship Y could have?

Answer 3

Y_i = α + ε_i for i = 1, …, N
Here these is a constant unknown population parameter and random deviation from this parameter
E(Y_i) = α and the estimator for α and Y are a and Ŷ respectively but Ŷ_i = a in this case

Question 4

What is the sum of squared residuals for Y_i = α + ε_i?

Answer 4

SSR = Σ_i(Y_i - a)² = Σ_ie_i²
e_i = Y_i - Ŷ_i = Y_i - a

Question 5

What is the least squares estimator for Y_i = α + ε_i?

Answer 5

From the criterion function min_aSSR we can find the solution using ∂SSR/∂a = Σ_i∂e_i²/∂a which by the chain rule is Σ_i∂e_i²/∂e_i ∂e_i/∂a = -2Σe_i
The FOC sets this to zero so 0 = ΣY_i - Σa so a = 1/n ΣY_{i = Ȳ
Note that Ȳ is a function of random variable Y and so is also a random variable with a sampling variance}

Question 6

What is the iterated expectation?

Answer 6

Converting a conditional expectation to an expectation
E_X(E(Y|X)) = E(Y)
For continuous X, this comes from ∫_XE(Y|X=x)f_x(x)

Question 7

What is a least squares line?

Answer 7

An estimate of a linear relationship between two variables
Ŷ = a + bx where Ŷ is a predicted value paired with a particular x, a is the estimated intercept, and b is the estimated slope

Question 8

What is one assumption imposed by an OLS estimate?

Answer 8

Equal variances

Question 9

How do you find the estimators for slope and intercept of a least squares line?

Answer 9

min_{a, b} SSR
FOCs are ∂SSR/∂a = 0 and ∂SSR/∂b = 0 so for a this comes out to a = Ȳ - bX̄ (work through in your head), for b we have 0 = ΣX_ie_i = ΣXY - ΣaX - ΣbX² so ΣXY = Σ(Ȳ - bX̄)X + ΣbX² which can be expanded and rearranged to get ΣXY - ȲΣX = b(ΣX - X̄ΣX) and ΣX = nX̄ so b = S_XY/S_X² (sample covariance over sample variance)
a and b are random variables with expectation equal to the population parameters

Question 10

How can the slope estimator be found from Cov(Y, X)?

Answer 10

Cov((α + βX + ε), X) = Cov(α, X) + βCov(X, X) + Cov(X, ε) but last term assumed to be zero so β = Cov(X, Y)/Var(X) and the estimator b = 1/(n-1)Σ(Y - Ȳ)(X - X̄) / 1/(n-1)Σ(X - X̄)²

Question 11

What are the population and sample conditional expectation functions from OLS?

Answer 11

E(Y|X) = α + βX and E(Ŷ|X) = a + bX

Question 12

What relationship between Y and X does a linear model impose?

Answer 12

Constant ∂Ŷ/∂X_i
Not always optimal

Question 13

What is the relationship between the slope estimator and the sample correlation coefficient?

Answer 13

b = rS_Y/S_X