Lecture 8 - Gradient Methods

Question 1

What type of space affect optimisation (qualties of Hypothesis Space)

Answer 1

• Continuous: No sudden jumps; gradient methods work well.
• Discontinuous: May break gradient methods.
• Differentiable: Needed for gradient descent.
• Non-differentiable: Use direct methods instead.
• Low modality: Few minima (easier to optimize).
• High modality: Many local minima (harder).
• Pathological: Strange landscapes, e.g., spiky or flat plateaus.

Question 2

Why is it important to know the affects of spaces?

Answer 2

Because gradient descent assumes a nice, smooth bowl. If the space is bumpy or weird, the method may fail or converge slowly.

Question 3

What do derivatives tell us?

Answer 3

• First derivative f′(x): tells you the slope — use it to go up or down.
• Second derivative f′′(x): tells you the curvature — use it to judge how fast slope is changing.

Question 4

What questions do derivatives answer?

Answer 4

1. Where is the slope 0? ⇒ potential extrema (stationary points)
2. Is the point a min, max, or saddle?
3. Which way should I move for fastest descent?

Question 5

What is Gradient Descent and its rule?

Answer 5

Gradient descent is an iterative optimization algorithm used to find the minimum of a function.
It works by moving in the direction of the negative gradient, which is the direction of steepest decrease in the function.

x <- x - alpha * f’(x)

Question 6

What is Gradient Ascent and its rule?

Answer 6

Gradient ascent is the same idea, but instead of minimizing, it maximizes the function.
You move in the direction of the gradient — where the function increases most quickly.
x <- x + alpha * f’(x)

Question 7

What is the Gradient Ascent/Descent Algorithm process?

Answer 7

REFER TO SLIDES FOR BREAKDOWN

Question 8

What is the Stopping Criteria of Gradient Ascent/Descent?

Answer 8

You stop when:
* f′(x)=0: The slope is flat — potential min or max.
* ∣f′(x)∣<ϵ: Close enough to flat.
* Time or iteration limit reached.

Question 9

What do you need to consider about the Stopping Criteria of Gradient Ascent/Descent?

Answer 9

• A zero gradient (f′(x)=0) doesn’t always mean you’ve found a minimum
• Could be:
  ○ Maximum (gradient ascent)
  ○ Minimum (gradient descent)
  ○ Saddle point (neither — it’s flat but unstable)
• How do we tell the difference?
  ○ Use the second derivative, f′′(x)

Question 10

How do you determine if its a local min or local max?

Answer 10

f’(x) = 0, f’‘(x) > 0 (minimum)
f’(x) = 0, f’‘(x) < 0 (maximum)
f’(x) = 0, f’‘(x) = 0 (inconclusive)

Question 11

What is the Ideal Case of Gradient Ascent/Descent?

Answer 11

○ The function is smooth and differentiable
§ No jumps, kinks, or flat regions
§ Gradient exists everywhere
○ It has a single global minimum or maximum
§ No local minima or maxima to get stuck in
§ Gradient descent will always find the right answer
○ The gradient behaves predictably
§ Far from the minimum → big slope → big step
§ Close to the minimum → small slope → small step
This means you naturally slow down as you approach the minimum → smooth convergence.

Question 12

Why does the Ideal Case for Gradient Ascent/Descent matter??

Answer 12

• No overshooting
• No weird local optima
• No need to adjust α much
• Convergence is fast and stable

Question 13

What is Derivative Step Size?

Answer 13

• When you’re far from the minimum, the slope (derivative) is steep → the gradient is large → you take bigger steps
• When you’re close to the minimum, the slope flattens out → the gradient is small → you take smaller steps

Question 14

Why is Derivative Step Size helpful?

Answer 14

Because in many functions (especially quadratics), the gradient acts like a natural brake:
- Far away? → move quickly
- Close to the target? → slow down automatically
- This helps avoid overshooting the minimum
Hence, the derivative acts like a natural step-size scaler when α=1.

Question 15

What is Rayleigh Distribution Case?

Answer 15

• A Rayleigh distribution is asymmetric:
  ○ Steep on one side
  ○ Flat on the other
• Where the Gradient descent might:
  ○ Take small steps where slope is flat
  ○ Overshoot where slope is steep

Question 16

Why is it a problem if the Gradient takes small steps or overshoots in Rayleigh Distribution

Answer 16

Why is this a problem?
- Poor convergence, unpredictable behaviour. Choosing α becomes difficult.

Question 17

What are the Tradeoffs when choosing Step Size for Rayleighs distribution?

Answer 17

Choosing Step Size
Trade-offs:
- Too small:
○ Converges slowly
○ Wastes computation
- Too large:
○ Overshoots the minimum
○ Can oscillate or diverge

Question 18

What is the Newton-Raphson Method?

Answer 18

The Newton-Raphson Method is a fast, iterative algorithm used to find:
1. The roots (zeros) of a function (where f(x)=0), or
2. The optima of a function (where f′(x)=0)

Question 19

How do you find the roots in Newton-Raphson Method

Answer 19

xn + 1 = xn - ( ((f’(x)) / (f’‘(x)) )

Question 20

How is Newton-Raphson used for Optimisation?

Answer 20

You apply the formula to the derivative to find the minima or maxima

Question 21

Newton-Raphson Algorithm

Answer 21

REFER TO SLIDES

Question 22

What is Smoothness?

Answer 22

Smoothness refers to how “nice” or well-behaved a function is, especially in terms of:
* Continuity (no jumps)
* Differentiability (has a slope)
* Second-order differentiability (has curvature)
Smoothness is based off classes -REFER TO NOTES

Question 23

Why is Smoothness important in Newton-Raphson

Answer 23

It requires a C^2 class where the function is continuous, first and second derivative exists and those derivatives are also continuous

Question 24

What are the limitations of Newton-Raphson?

Answer 24

Requires second derivative - not all functions a twice differential
Can diverge - if stating points is too far from root or second derivaive is too small or 0
Doesnt guarantee global minimum - only finds local optimum which could a min, max or saddle point

Question 25

What are the limitations of Gradient Ascent/Descent?

Answer 25

Needs careful choice of step size - too big it will overshoot, too small slow convergence May get stuck in local minima Slows down near the minimum - as gradient gets too small

Question 26

What is a Local Optimum?

Answer 26

A local optimum is a point where the function is better than its immediate neighbours.

Question 27

What is a Global Optimum?

Answer 27

A global optimum is the absolute best value across the entire domain.

Question 28

What is the general understanding of Global Optimum?

Answer 28

- There is no general-purpose algorithm that can guarantee finding the global optimum in all cases. - Although we can still find good solutions, even if we can't guarantee the best one. ○ Why is this important? * Because in the real world, we: □ Rarely need perfect solutions □ Often just want “good enough”, quickly □ Work in messy domains with weird curves, noise, or constraints

Question 29

What are the approaches used to find the good solutions?

Answer 29

* We use: □ Gradient descent with restarts □ Stochastic methods (e.g., simulated annealing, genetic algorithms) □ Heuristics or metaheuristics (e.g., greedy strategies, random sampling) * These don't guarantee global optima, but they explore the space better and often find excellent local optima.

Question 30

Q: Why can’t gradient methods guarantee a global optimum?

Answer 30

Because they follow the gradient, which only leads to a local minimum or maximum. In non-convex functions, there may be multiple optima. Without exploring the entire space (which is often infinite or too large), there’s no way to know if a better one exists elsewhere.

Question 31

What are the fundamental limitations in optimisation (especially when trying to find the global optimum)

Answer 31

Non-Enumerable Domains - the domain is infinite, this is an issues as you cant test every possible value Huge Search Spaces - So many possibilites that you cannot search them in reasonable time, this is an issue as you might need too long which defeats the purpose

Question 32

What is Gradient Ascent with Restarts?

Answer 32

This method is an enhancement of basic gradient ascent. It helps address the problem of getting stuck in a local maximum. - Instead of trusting just one run of gradient ascent, you try multiple random starts and keep track of the best solution you find. - This increases your chances of getting closer to the global maximum.

Question 33

Why use Gradient Ascent with Restarts?

Answer 33

* Basic gradient ascent is deterministic: it follows the same path from the same start point. * If it starts near a local maximum, it may stop there and miss better peaks. * Restarting from different points gives a better chance to explore more of the space.

Question 34

Gradient Aescent with Restarts Algorithm

Answer 34

REFER TO NOTES

Question 35

What is the Stopping Criteria for Gradient Ascent with Restarts?

Answer 35

* The inner loop stops when ∣∇f(x)∣<ϵ→ slope is nearly zero (flat) * The outer loop stops when: ○ You’ve run a fixed number of iterations ○ Or a time/resource limit is reached

Question 36

Q: Why does adding restarts improve Gradient Ascent?

Answer 36

Gradient ascent can get stuck in local maxima. By restarting from different random positions, the algorithm explores multiple regions of the search space. Comparing function values across all runs helps identify the best peak found, even if it’s not the global optimum.

Question 37

Q: What extra requirement does gradient ascent with restarts introduce?

Answer 37

You must be able to compute the actual function value f(x) (not just its gradient) to compare and retain the best result.

Question 38

How does this method (Gradient Ascent with Restarts) try to overcome the challenge of local optima?

Answer 38

Through multiple random initializations and tracking the best result, it reduces the chance of getting stuck at a poor-quality local maximum.

Question 39

What are the Challanges in Optimisation?

Answer 39

Large, Space Search Spaces Local Optima Traps Plateaus Non-differential points Valleys in High Dimentions

Question 40

Large, Space Search Spaces - Challanges in Optimisation

Answer 40

Space of possible solutions is too big - This is a problem as you may be wasting time, Gradient method may converge too slowly, cant check every point

Question 41

Local Optima Traps - Challanges in Optimisation

Answer 41

Points looks like a minimum or maximum but its not the global best - Problem as GD or NR methods stop at slope = 0, could be stuck in a local dip

Question 42

Plateaus - Challanges in Optimisation

Answer 42

A flat region where the slope is zero or nearly zero everywhere - Problem as GD relise on the slope to know where to move, very slow or no progress

Question 43

Non-differential points - Challanges in Optimisation

Answer 43

No derivative at this point - Problem as gradient based function need the function to be differential, the slope is undefinedm breaking the behaviour

Question 44

Valleys in High Dimentions - Challanges in Optimisation

Answer 44

Narrow curved valleys where the gradient, points off the correct direction or changes slowly along one axis and slow along the other - Problem as GD sutrggles as it moves in one direction at a time, leads to slow inefficient convergences