Linear Regression

Question 1

What is linear regression?

Answer 1

Linear regression assumes a linear functional relationship between the predictor and target variables

Question 2

What is the formula for linear regression?

Answer 2

y = f(x,w) + ∈

Question 3

Why can we use linear and non-linear basis functions for linear regression?

Answer 3

Because our model remains linear w.r.t w

Question 4

What does a learned regression model do?

Answer 4

A learned regression model predicts y for new values of x

Question 5

What is Ordinary Least Squares (OLS)?

Answer 5

Ordinary Least Squares finds the weights that minimises the sum of squared residuals

Question 6

What is the formula for OLS?

(also known as squared error)

Answer 6

R(w) = Σ ri^2
= Σ(yi - ŷi(xi,w))^2
= Σ(yi - wTΦ(xi))^2

Question 7

What is the regression function?

Answer 7

ŷ(xi,w) = Σ wjΦj(xi) = wTΦ(xi)

Question 8

What do we get when we minimise R(w) w.r.t w to estimate Wols?

Answer 8

Σ yiΦT(xi) = wT(Σ Φ(xi)ΦT(xi))

This can be expressed in matrix-vector form by defining the design matrix Φ

Question 9

What is the normal equation for solving Wols?

Answer 9

ΦTy = ΦTΦw
Wols - (ΦTΦ)^-1 ΦTy

(also known as computing the pseudoinverse of the design matrix)

Question 10

What is a consequence of using the inversion in the normal equation?

Answer 10

Numerical inversions of matrices can be computationally expensive if the matrix is large

Question 11

What is the in-sample error?

Answer 11

Ein(h) = 1/N Σ (h(x(n)) - y(n))^2

Question 12

What is the hypothesis for linear regression?

Answer 12

h(x) = Σ wixi = wTx