Gradient Descent Flashcards
What is Gradient Descent?
Gradient Descent is an iterative solution that incrementally steps towards an optimal solution.
Highlevel steps of Gradient Descent
- Start with an initial guess
- Improve the guess one step at a time..
- …until it finds an optimal solution or reaches the maximum number of steps.
In linear regression, Gradient Descent estimates ___________________ and ________________ to minimize ____________________
- Intercept
- Slope
- Sum of squared residuals
Loss or _________ function
cost
What is Loss function in Linear regression
Since the goal is to minimize SSR, it’s a type of Loss or cost function.
What is Loss or cost function
The terms loss or cost function refer to anything we want to optimize when we fit a model to data. In linear regression, the cost/loss function is SSR/MSE. That being said, some people refer to Loss function when the function is applied to only one data point. And Cost function when the function is applied all of the data.
What is tangent line
Tangent line touches the curve at a single point and doesn’t intersect with it.
Derivation of Gradient descent
Intercept and slope are called _____________ in machine learning
Parameters
How do you calculate step size?
step size = derivate * learning rate
What is learning rate?
The learning rate prevents us from taking step that are too big and skipping past the lowest point in the curve.
How is learning rate determined for Gradient descent?
For Gradient Descent, the learning rate is determined automatically. It starts with relatively large and gets smaller with every step.
Apart from default learning rate in gradient descent, what approach can you use?
You can use cross validation to determine good learning rate.
What is cross validation?
Cross validation is a technique used in machine learning to evaluate the performance of a model on unseen data.
What is the relationship between derivative and learning rate
The magnitude of the derivative is proportional to how big of a step we should take toward the minimum. The +/- tells us the direction.
