Test 5

Question 1

5 Step Training Loop

Answer 1

1. Generate Prediction - model(x, y)
2. Compute Loss - loss(y, y~)
3. Zero Gradient Parameters (optimizer.zero_grad())
4. Compute Gradients (loss.backward())
5. Change weights - optimizer.step()

Question 2

What does the feature vector project onto the weight space?

Answer 2

Only the POSITIVE projection of z onto the w-space

Question 3

What are the 7 steps that a NN layer performs

Answer 3

Rotation, Reflection, Scaling, Rotation, Reflection, Bias, Positive Part

Question 4

1989

Answer 4

Yann Lecunn Introduces CNNs which perform dimensional expansion without blowing up the number of weights, allowing us to compute the feature vector from an image

Question 5

Three Properties of a Wave/Signal

Answer 5

Stationarity - Fluctuation of a Wave Does not Change Much - You will see similar patterns

Locality - Things close in domain-space are more likely to be correlated

Compositionality - The overall information is transmitted through the integration of the parts

Question 6

How is stationarity represented in CNNs?

Answer 6

Parameter Sharing

Question 7

1997

Answer 7

Jordan Schmidhuber introduces the long term short memory cell, a dynamical system (RNN) that is able to model long term dependencies

Question 8

If you have multiple partial derivatives across each time step for the same weights, how do we resolve this in an RNN?

Answer 8

We sum the partials