Optimizers

Question 1

What is optimization?

Answer 1

determines the algorithms we will use to vary our model’s parameters

Question 2

What is a type of optimization algorithm?

Answer 2

1. Gradient decent - slow
2. SGD - Stochastic Gradient Descent

Question 3

What is the difference between the local minimum and global minimum?

Question 4

How do we extend the GD and SGD?

Answer 4

using momentum

Question 5

What is momentum?

Question 6

Is the Alpha of momentum a hyper parameter?

Answer 6

Yes

Question 7

What is the learning rate?

Answer 7

eta - (greek)

Small enough - so we gently descend through the loss function, instead of oscillating wildly around the minimum and never reaching it or diverging to infinity

Big enough - so we reach the optimization in a rational amount of time

Question 8

What are learning rate schedules?

Answer 8

Best of both worlds. Small enough and Big enough

it causes the loss function to converge much faster

Question 9

What are two examples of Adaptive Learning Rate Schedules?

Answer 9

AdaGrad and RMSProp

AdaGrad = Adaptive Gradient Algorithm

RMSProp = Root Mean Square Propagation

Adam = Adaptive moment estimation