Backpropagation

Question 1

Hadamard product

Answer 1

Question 2

Finite differences

And limitations of

Answer 2

Can be a poor approximation if F is highly non linear

Question 3

Symbolic differentiation

Limitations?

Answer 3

A symbolic expression for F’(x) is derived using computer algebra, instantaneously applicable to any argument x

Problematic if F is a complicated function (eg composite of large number of functions)

Question 4

Automatic differentiation

Answer 4

/algorithmic differentiation

Provides way to efficiently compute F’(x) The exactly, but for a chosen x only

BACKPROP is a special case of (backward mode) algorithmic differentiation

Question 5

One line description of backprop

Answer 5

Computes gradient of empirical risk for a FNN

Question 6

Per sample gradient

Answer 6

Question 7

Introduce notation (required) to define back prop

Answer 7

Question 8

Chain rule

Answer 8

Question 9

Back prop procedure

Answer 9

Question 10

Prove back prop

Answer 10

Question 11

Forward pass

Answer 11

(3.13 is back prop procedure)

Question 12

Formula for adjoint

Answer 12