Logistic Regression

Question 1

Steps in logistic Regression

Answer 1

1. Desing an appropriate function for y based on x
2. and build a probabilistic model for p(y|x)
3. Determine parameters of the function

Question 2

Step 1

Answer 2

Determine an appropriate function that maps the data to a class

y_pred=f(x)=σ(z) with z=w^Tx

Question 3

Step 2

Answer 3

For example p(y=1|x;theta)=f(x)
p(y=0|x;theta)=1-f(x)

Or as Bernoulli

Question 4

Step 3

Answer 4

Determine parameters of the function
-> maximize the likelihood function

L(theta)=p(Y|X,theta)

And build the log likelihood

Question 5

What is a problem of the log-likelihood derivative?

Answer 5

It does not lead to a closed form solution.

Solution: iterative non-linear optimization with gradient descent

theta = argmin L(theta)
theta := theta-alpha*l(theta)

Question 6

What training algorithm is being used?

Answer 6

Stochastik gradient Descent

function SGD(D, alpha):
Initialize theta with a small random value
repeat
for d=1 to m do:
theta <- theta + alpha*delte_theta
end for
until change of theta is small
return theta
end function