Linear Regression

Question 1

What are basis functions? What they possibilitate?

Answer 1

Are functions that transform the input

They are used to extend linear models to non-linear relationships

Question 2

What is the Ordinary Least Square? When is it most suitable to use?

Answer 2

It is a method that calculates the optimal weight vector by multiplying the inverse of the multiplication of the transpose of the basis by the basis by the transpose of the basis and the target vector.

It derives the optimal weights analytically using normal equations.

It yields the exact global optimum when the error function is quadratic and convex .

It is most suitable to smaller datasets.

Question 3

How can we solve a regression problem for large scale problems?

Answer 3

For large scale problems where the closed-form solutions is impractical, iterative methods such as stochastic gradient descent are discussed.

Question 4

Why the closed-form solution is not suitable for large problems?

Answer 4

Because it requires the computation of the inverse of the multiplication of the transpose of the basis by the basis what is computationally expensive with larga datasets (roughly O(M^3)

Also it assumes that that same matrix is nonsingular (invertible), if this condition fails, the closed-form solution might not exist or require regularization.

Question 5

What is the central goal when using maximum likelihood for solving linear regressions? What does the likelihood function measure?

Question 6

What are the steps taken to maximize the likelihood that a given set of parameters generated the data we have?

Question 7

What is the size of the penalties for Ridge and Lasso? Which one can be solved with a closed form solution? Which one produce sparse models?

Answer 7

Proportional to square of weights

Proportional to absolute value of weights

Question 8

What is the basic assumption done in the Bayesian approach to linear regression? What does this possibilitates?

Question 9

What does it means to consider the weights as random variables?

Question 10

Give a general framework starting from the prior distribution of the weights up to the prediction moment.

Question 11

How does the Bayesian method related to regularizatiin?