Linear Regression

Question 1

What is supervised learning?

Answer 1

A learning paradigm where a model is trained on input-output pairs (x_i, y_i) to learn a mapping from inputs to outputs.

Question 2

How is the linear regression model expressed using basis functions?

Answer 2

f(x) = ∑ₖ βₖ φₖ(x), where φₖ are basis functions (e.g., φ₀(x) = 1, φ₁(x) = x).

Question 3

What loss function does linear regression use and why?

Answer 3

Mean Squared Error: L = (1/n) ∑ᵢ (yᵢ − f(xᵢ))²; penalizes larger errors and is differentiable.

Question 4

What is the normal equation for the closed-form solution?

Answer 4

β = (Φᵀ Φ)⁻¹ Φᵀ y, where Φ is the design matrix of basis functions.

Question 5

How does gradient descent optimize parameters?

Answer 5

β_new = β_old − rate × ∂L/∂β, thus moving each step in the negative‐gradient direction to reduce the loss.

Question 6

What is the gradient descent update rule in linear regression?

Answer 6

β_new = β_old – rate * (–2 * Φ^T * y + 2 * Φ^T * Φ * β_old), Step in the negative‐gradient direction of the MSE loss.

Question 7

How does learning rate affect gradient descent convergence?

Answer 7

A small learning rate yields slow convergence; a large learning rate can cause oscillation or divergence.

Question 8

How is the design matrix Φ?

Answer 8

Φ = np.vstack((np.ones(n), x)), stacking a row of ones (for the intercept) on top of the feature row.