9 - The Man Who Set Back Machine Learning (Not Really)

Question 1

What is deep learning?

Answer 1

The process of training neural networks that have three or more layers (one input layer, one output layer, and one or more hidden layers).

Question 2

Who is George Cybenko?

Answer 2

A professor of engineering at Dartmouth College known for his work on neural networks.

Question 3

What significant event occurred in 2017 related to deep learning?

Answer 3

A summer school on deep learning in Bilbao, Spain, attended by nearly thirteen hundred people.

Question 4

What is the universal approximation theorem?

Answer 4

A theorem that shows a neural network with just one hidden layer, given enough neurons, can approximate any function.

Question 5

What did Cybenko’s landmark paper demonstrate?

Answer 5

It proved that a neural network with one hidden layer can approximate any function.

Question 6

What did Minsky and Papert’s book ‘Perceptrons’ conclude?

Answer 6

Single-layer neural networks have limitations and multi-layer networks are likely useless.

Question 7

What is backpropagation?

Answer 7

A training algorithm for neural networks that allows for the training of multi-layer networks.

Question 8

Who were the authors of the seminal 1986 backpropagation paper?

Answer 8

David Rumelhart, Geoffrey Hinton, and Ronald Williams.

Question 9

What is the role of a single-layer perceptron?

Answer 9

It takes input values, calculates weighted sums, and produces an output based on a thresholding function.

Question 10

What is the perceptron training algorithm used for?

Answer 10

To train a single-layer neural network by finding optimal weights and biases.

Question 11

What differentiates a deep neural network from a single-layer network?

Answer 11

A deep neural network has multiple weight matrices due to the presence of hidden layers.

Question 12

True or False: The perceptron training algorithm can be used for networks with hidden layers.

Answer 12

False.

Question 13

What does training a neural network involve?

Answer 13

Finding optimal values for the weight matrices to approximate a desired function.

Question 14

Fill in the blank: Cybenko’s work is often associated with _______.

Answer 14

delaying deep learning by twenty years.

Question 15

What can a function represented by a neural network achieve?

Answer 15

It can represent a decision boundary or perform regression.

Question 16

What is an activation function?

Answer 16

A function that determines the output of a neuron based on its input.

Question 17

What is the sigmoid activation function?

Answer 17

A smooth function that transitions from almost 0 to almost 1.

Question 18

What type of neuron did Cybenko use in his proof?

Answer 18

A nonlinear neuron based on the sigmoid activation function.

Question 19

What is the significance of having multiple weight matrices in a network?

Answer 19

It allows the network to be classified as a deep neural network.

Question 20

What does the term ‘AI winter’ refer to?

Answer 20

A period of reduced funding and interest in artificial intelligence research.

Question 21

What is a hidden layer in a neural network?

Answer 21

A layer of neurons that is not directly exposed on the output side.

Question 22

What is the output of a bipolar neuron?

Answer 22

+1 or -1.

Question 23

What activation function is used in Cybenko’s neurons?

Answer 23

The sigmoid activation function, a(z) = σ(z)

The sigmoid function smoothly transitions from almost 0 to almost 1.

Question 24

What does the equation z = wx + b represent in the context of a neuron?

Answer 24

The weighted sum of the inputs plus the bias term

Here, z is calculated using the weight vector w and bias b.

Question 25

How can the shape and position of the sigmoid function be controlled?

Answer 25

By changing the values of w (weights) and b (bias) ## Footnote This affects the steepness and midpoint of the sigmoid curve.

Question 26

What is the output equation for the neuron?

Answer 26

y = s(z) ## Footnote s(z) is the sigmoid activation function applied to z.

Question 27

What is the significance of the output neuron in a hidden layer network?

Answer 27

It performs a linear combination of the outputs of the hidden neurons ## Footnote This involves multiplying each output by a weight and summing them.

Question 28

What is the main goal of Cybenko's analysis on neural networks?

Answer 28

To prove that a summation of hidden neurons can approximate any desired function f(x) ## Footnote This is dependent on having enough hidden neurons.

Question 29

What does the term 'one-dimensional version' refer to in Cybenko's analysis?

Answer 29

Both input and output vectors have only one element each ## Footnote This simplifies the understanding of the network's operations.

Question 30

How does the output neuron generate an approximately rectangular output?

Answer 30

By performing a linear combination of the outputs of two hidden neurons ## Footnote The weights used can be positive or negative, affecting the final shape.

Question 31

What happens when the number of sigmoidal neurons is increased?

Answer 31

The approximation of the desired function improves ## Footnote More neurons allow for better representation of complex functions.

Question 32

What is the relationship between weights, biases, and the outputs of hidden neurons?

Answer 32

Weights and biases determine the shape and position of the hidden neuron outputs ## Footnote This affects the final output through linear combinations.

Question 33

What did Cybenko prove about neural networks in 1988?

Answer 33

That a network with two hidden layers can approximate any function ## Footnote He believed it should also be possible with just one hidden layer.

Question 34

What is the significance of considering functions as vectors?

Answer 34

It allows us to analyze functions in higher-dimensional spaces ## Footnote Functions can be represented as points in infinite-dimensional spaces.

Question 35

What does the term 'vector space' refer to?

Answer 35

A collection of vectors, matrices, and functions that satisfy certain properties ## Footnote These objects can be manipulated mathematically.

Question 36

What was Cybenko's proof method?

Answer 36

Proof by contradiction ## Footnote He assumed that a neural network could not approximate all functions and showed this assumption led to a contradiction.

Question 37

What misconception did Cybenko's proof create among researchers?

Answer 37

That only one hidden layer was necessary for neural networks ## Footnote This led to a focus on shallow networks, delaying the deep learning revolution.

Question 38

What factors contributed to the revolution in deep learning around 2010?

Answer 38

Increased number of hidden layers, massive training data, and computing power ## Footnote These elements were not widely available in the 1990s.

Question 39

Fill in the blank: Cybenko believed a neural network with one hidden layer could approximate _______.

Answer 39

any function ## Footnote This belief was significant for the development of neural network theory.

Question 40

What significant development in deep learning occurred in 2010?

Answer 40

Researchers began to increase the number of hidden layers in neural networks beyond one. ## Footnote This marked a pivotal shift in the capabilities of deep learning.

Question 41

What was Cybenko's contribution to neural networks?

Answer 41

Cybenko proved the approximating properties of neural networks in 1989. ## Footnote His work laid the groundwork for future advancements in deep learning.

Question 42

What two key ingredients were necessary for the deep learning revolution to take off in the 2010s?

Answer 42

* Massive amounts of training data * Computing power ## Footnote These factors were not available in the 1990s.

Question 43

What did Cybenko speculate about the number of neurons required for approximation?

Answer 43

He speculated that astronomical numbers of terms would be required for most approximation problems. ## Footnote This was influenced by the curse of dimensionality in multidimensional approximation theory.

Question 44

What is the curse of dimensionality?

Answer 44

The curse of dimensionality refers to the phenomenon where the feature space becomes increasingly sparse as dimensionality increases, making it difficult for models to generalize. ## Footnote This concept poses challenges in statistics and approximation theory.

Question 45

How do deep neural networks challenge traditional machine learning theories?

Answer 45

They are not as susceptible to the curse of dimensionality and do not overfit the data as expected despite having massive parameters. ## Footnote This challenges existing beliefs about model complexity and data fitting.

Question 46

What algorithm allowed researchers to start training deep neural networks?

Answer 46

Backpropagation. ## Footnote This algorithm is essential for optimizing the weights in neural networks during training.