Final Deep Learning

Question 1

1943

Answer 1

McCulloch/Pitts - Binary Neuron activation - they activate above a threshold [s >0]; allowed us to model neurons as non-linear functions

Question 2

1948

Answer 2

Wiener - Feedback Cycles in the brain/with neurons - allowed us to give and get feedback for artificial neurons

Question 3

1949

Answer 3

Hebb - NeuroPlasticity/Synaptic Adaptability - Taught us how to adjust/train neurons

Question 4

1957

Answer 4

Rosenblatt - Single Layer Perceptron - [wTx] > 0 and learning rule; created first artificial neuron

Question 5

Learning Rule for single layer perceptron

Answer 5

W1 <- w1 - w1f(x)

Question 6

Consider tossing a coin where p = probability of heads

Write P(y) in terms of Q(p)^(R(y))

Answer 6

P^y * (1-p)^(1-y)

Question 7

For a single layer perceptron, how do you calculate the probability of a binary output?

Answer 7

Sigmoid(wTf(x))

Question 8

For a binary prediction, write P(y|x) in polynomial multiplication form

Answer 8

sigmoid(wtf(x))^y*(1 - sigmoid(wtf(x))^(1-y)

Question 9

Give the formula for H(y, y~) and explain why it’s important

Answer 9

Binary cross entropy - H(y, y~) is called cost because it tells us how far off the prediction is

H(y, y~) = ylog(y~) + (

Question 10

Give the formula for H(y, y~) and explain why it’s important-

Answer 10

Binary cross entropy - H(y, y~) is called cost because it tells us how far off the prediction is

H(y, y~) = ylog(y~) + (1-y)(1-log(y~))

Question 11

L(w, x, y) = ?

Answer 11

ylog(1 + exp(-s)) + (1-y)(1-log(1 + exp(s))

Question 12

Soft plus

Answer 12

Log(1 + exp(s))

Question 13

L(w, x, y) with soft plus

Answer 13

ysoftplus(-s) + (1-y)softplus(s)

Question 14

Rewrite loss in terms of log likelihood

Answer 14

substitute in ynlog(p(y=1|x)) + (1-yn)log(P(y=0|x))

Question 15

How do we recover the perceptron learning rule w <- w + (y - y~)f(x) from minimizing the loss function of the perceptron

Answer 15

When you take the derivative of the sigmoid (wtfx), you get (y^ - y)fx

Question 16

Let y = onehot_k(y), the y-th column of I_k, the identity matrix of size K. Give the expression for the scalar P(y|x)

Answer 16

P(y|x) = exp(Bs_y)/(sum(exp(Bs_k)

Question 17

Probability of dataset D with a soft multiclass perceptron

Answer 17

product of y^t, y~ for all yn

Question 18

Cross Entropy for a soft multiclass perceptron

Answer 18

H(y, y~) = -log[y^ty~]

Question 19

Rewrite cross entropy loss using softmax function

Answer 19

softmax(wtf) - wy^Tf

Question 20

multiclass perceptron learning rule

Answer 20

W <- w + (y - y~)fT(x)

Question 21

1969

Answer 21

Seymour/Minsky - Seymour Report - Perceptrons will not converge with non-linearly separable data, killing hopes for AI

Question 22

1973

Answer 22

UK National Science Council - Lighthall Report - AI Research is not promising, causing an AI Winter

Question 23

1982

Answer 23

Hopfield - Proposed Hopfield Net, an associative memory that uses the McCulloch and Pitts binary Neuron and minimizing energy

Question 24

1986

Answer 24

Williams;Hinton - Backpropagation/GD w<- w + lr*dLoss/dw, as well as stochastic hopfield nets with hidden units. Allowed us to easily calculate the gradient of NNs with hidden layers, creating deep networks

Question 25

How did we get out of the first AI winter?

Answer 25

Widrow, who also introduced the adaptive linear neuron, changed AI research names to Adaptive Filters

Question 26

1989

Answer 26

Yann Lecunn invents CNNs, now we're able to process images with less parameters allowing big inputs with low computation

Question 27

1997

Answer 27

Jordan Schmidhuber introduces the LSTM, allowing us to model long-term dependencies in sequences

Question 28

give the formal definition of a signal over X

Answer 28

X = {x^(p): omega -> R^c, w -> x^(p)(w)} where omega is the domain, c is the channelsh

Question 29

What are the three properties of natural signals exploited by CNNs and how do they do so?

Answer 29

Stationarity - parameter sharing Locality - kernel preserves spatial awareness Compositionality - The transformed output is the whole input put together