11 - The Eyes of a Machine

Question 1

Who are the co-founders of the Department of Neurobiology at Harvard?

Answer 1

David Hubel and Torsten Wiesel

They were awarded the Nobel Prize in Physiology or Medicine in 1981.

Question 2

What was the main focus of Hubel and Wiesel’s research?

Answer 2

The visual system of cats

Their work involved mapping the visual cortex and understanding neural responses to visual stimuli.

Question 3

What invention did Hubel create in 1957 for recording neuron activity?

Answer 3

A tungsten electrode

This electrode was preferred over glass micropipettes and steel electrodes for its durability.

Question 4

What anesthetic was used on cats during Hubel and Wiesel’s experiments?

Answer 4

Intraperitoneal thiopental sodium

This was administered to keep the cats under anesthesia during the experiments.

Question 5

What is a ‘visual field’?

Answer 5

The region in front of us that our eyes are sensitive to

It changes as we move our eyes and focus on different objects.

Question 6

Define ‘receptive field’.

Answer 6

The portion of the visual field that triggers a single neuron

The size of the receptive field can vary significantly among different neurons.

Question 7

What are retinal ganglion cells?

Answer 7

Neurons that monitor the image on the retina

They are the first layer of neurons receiving inputs from the retina.

Question 8

What is the function of a ‘simple cell’ in the visual cortex?

Answer 8

It fires only when all its connected retinal ganglion cells fire together

Simple cells are sensitive to specific orientations of edges.

Question 9

What does a ‘complex cell’ do?

Answer 9

It fires in response to an edge regardless of its position in the receptive field

This indicates spatial or translational invariance.

Question 10

What does ‘invariance’ refer to in the context of vision?

Answer 10

The ability to recognize stimuli regardless of their position or orientation

This includes translational and rotational invariance.

Question 11

What is a ‘hypercomplex cell’?

Answer 11

A cell that fires for an edge of a specific length and orientation

It can detect complex shapes and patterns.

Question 12

What was the name of the first neural network-based image recognition system?

Answer 12

Cognitron

Developed by Kunihiko Fukushima in 1975.

Question 13

How does the learning algorithm in the cognitron work?

Answer 13

It strengthens synaptic connections based on neuron firing activity

This is akin to Hebbian learning principles.

Question 14

What limitation did the cognitron have?

Answer 14

It was not translation invariant

This means it recognized patterns differently based on their position in the visual field.

Question 15

What architecture did the neocognitron adopt from Hubel and Wiesel’s work?

Answer 15

It includes S-cells and C-cells

These cells model simple and complex cells found in the visual cortex.

Question 16

What is the role of S-cells in the neocognitron?

Answer 16

They respond to specific features like vertical edges

S-cells feed into C-cells to indicate edge presence.

Question 17

What does the term ‘grandmother cell’ refer to?

Answer 17

A hypothetical neuron that fires when seeing a specific complex stimulus

This concept is often used to illustrate the idea of specific neural responses to familiar faces.

Question 18

What are C-cells in the context of the visual processing model?

Answer 18

C-cells respond to vertical edges in different patches of the visual field.

Question 19

What is the role of S-cells in the visual processing architecture?

Answer 19

S-cells collate outputs from C-cells to detect edges anywhere in the visual field.

Question 20

What does translation invariance refer to in the neocognitron?

Answer 20

The ability to detect patterns regardless of their position or distortions.

Question 21

Who developed the neocognitron and what was its significance?

Answer 21

Fukushima developed the neocognitron, which advanced pattern recognition in machines.

Question 22

What was the main limitation of the neocognitron’s training algorithm?

Answer 22

It adjusted only the weights of the S-cells and was cumbersome.

Question 23

What breakthrough did Yann LeCun achieve in the field of neural networks?

Answer 23

He developed the convolutional neural network (CNN) using the backpropagation algorithm.

Question 24

How did the meeting between LeCun and Fukushima impact the field?

Answer 24

Fukushima was surprised to learn that LeCun’s work was parallel to his own on the same topic.

Question 25

Who were the key figures in the debate about cognitive capabilities, and what were their positions?

Answer 25

Piaget believed in learning during development; Chomsky argued for innate capabilities.

Question 26

What was Papert's stance during the Piaget-Chomsky debate?

Answer 26

Papert supported Piaget, arguing that Chomsky underestimated the role of learning.

Question 27

What is a perceptron?

Answer 27

An early type of artificial neural network used to analyze learning and cognition.

Question 28

What was LeCun's key realization regarding learning algorithms?

Answer 28

A learning algorithm should minimize an objective function.

Question 29

What is the difference between a loss function and an objective function?

Answer 29

The objective function includes a regularizer to prevent overfitting, while the loss function does not.

Question 30

What is the purpose of a regularizer in machine learning?

Answer 30

To prevent overfitting by adding an extra term to the loss function.

Question 31

What was the focus of LeCun's doctoral work?

Answer 31

Developing a learning algorithm related to backpropagation for multi-layer neural networks.

Question 32

What did LeCun present at a conference in 1985?

Answer 32

A poorly written paper in French on his learning algorithm.

Question 33

What significant software did LeCun and Bottou create?

Answer 33

SN, which later evolved into Lush, an early neural network simulation tool.

Question 34

What major dataset did LeCun work with at Bell Labs?

Answer 34

Images of handwritten digits from the U.S. Postal Service.

Question 35

What is the convolution operation in the context of image processing?

Answer 35

An operation that combines two functions to produce a third, particularly for feature extraction.

Question 36

What are kernels in convolutional neural networks?

Answer 36

Small matrices used to filter images and detect features such as edges.

Question 37

What is the effect of convolution on the size of an image?

Answer 37

Convolution typically reduces the size of the output image.

Question 38

What is the stride in convolution operations?

Answer 38

The number of pixels by which the kernel moves across the image during convolution.

Question 39

What is the formula for calculating the output image size after convolution?

Answer 39

((i - k) / s) + 1, where i is the input size, k is the kernel size, and s is the stride.

Question 40

True or False: The original image size remains the same after applying a convolution operation.

Answer 40

False.

Question 41

What is the term used for the distance the kernel moves in a convolution operation?

Answer 41

Stride ## Footnote The stride can affect the size of the output image.

Question 42

How is the output image size calculated in a convolution operation?

Answer 42

(( i - k ) / s ) + 1 ## Footnote Where i is the input image size, k is the kernel filter size, and s is the stride.

Question 43

What is padding in the context of convolutional networks?

Answer 43

Dummy pixels added around the input image ## Footnote Padding can affect the output size and feature extraction.

Question 44

What is the output size when using a 3×3 kernel on a 28×28 image with a stride of 1?

Answer 44

26×26 ## Footnote This is calculated without padding.

Question 45

What represents the weighted sum of pixel values in a convolution operation?

Answer 45

The output of a neuron ## Footnote The weights correspond to the kernel values.

Question 46

How many neurons are needed for a 5×5 image with a 2×2 kernel and a stride of 1?

Answer 46

16 neurons ## Footnote This produces a 4×4 output image.

Question 47

Define receptive field in the context of convolutional neural networks.

Answer 47

The specific area of the image that a neuron responds to ## Footnote Each neuron has its own region of interest.

Question 48

What are simple cells and complex cells in the hierarchy of visual processing?

Answer 48

Simple cells respond to simple features and complex cells respond to compositions of those features ## Footnote This hierarchy was posited by Hubel and Wiesel.

Question 49

What is max pooling in convolutional neural networks?

Answer 49

An operation that outputs the largest pixel value from a region under the filter ## Footnote Max pooling reduces image size and increases the receptive field.

Question 50

What is the effect of max pooling on the size of an image?

Answer 50

Reduces the size of the image ## Footnote For example, a 4×4 image with a 2×2 filter and stride of 2 results in a 2×2 image.

Question 51

What is the purpose of backpropagation in training neural networks?

Answer 51

To calculate gradients and update weights ## Footnote This process minimizes the error between expected and actual outputs.

Question 52

What is the role of activation functions in neural networks?

Answer 52

To introduce non-linearity and enable backpropagation ## Footnote Activation functions must be differentiable.

Question 53

What are hyperparameters in the context of neural networks?

Answer 53

Parameters not learned during training that influence performance ## Footnote Examples include the number of layers, kernel sizes, and activation functions.

Question 54

What was LeNet used for in the banking industry?

Answer 54

To read and recognize digits on checks ## Footnote LeNet was one of the early successful applications of convolutional neural networks.

Question 55

What does the output layer of a neural network for digit recognition typically consist of?

Answer 55

10 neurons ## Footnote Each neuron corresponds to a digit from 0 to 9.

Question 56

What happens during stochastic gradient descent?

Answer 56

A subset of images is used for each pass through the network ## Footnote This is a method to optimize the learning process.

Question 57

What was a significant limitation of convolutional neural networks in the 1990s?

Answer 57

Lack of general-purpose software to build CNNs ## Footnote AT&T did not allow the distribution of their software open source.

Question 58

What algorithms were outperforming CNNs for low-resolution images in the 1990s?

Answer 58

Conventional techniques ## Footnote CNNs were still not widely adopted due to skepticism in the computer vision community.

Question 59

What hardware advancement in the 2000s significantly changed deep learning?

Answer 59

Graphical Processing Units (GPUs) ## Footnote GPUs were originally designed for rendering 3D graphics but proved vital for deep learning tasks.

Question 60

What dataset did Jürgen Schmidhuber's team use to train multi-layer perceptrons?

Answer 60

MNIST images ## Footnote They achieved low error rates of 0.35 percent using deep learning techniques.

Question 61

What was the name of the first massive CNN built by Hinton's lab?

Answer 61

AlexNet ## Footnote AlexNet demonstrated that conventional methods for image recognition could not compete.

Question 62

What was the purpose of vector maps in Hinton and Mnih's research?

Answer 62

To teach neural networks how to label pixels in aerial images ## Footnote Vector maps provided clear information about road locations.

Question 63

What programming interface allowed GPUs to be used for general-purpose tasks?

Answer 63

CUDA ## Footnote CUDA enabled engineers to perform tasks beyond just graphics acceleration.

Question 64

What breakthrough did Hinton's lab achieve in speech recognition?

Answer 64

Using CUDAMat to program deep neural networks ## Footnote This demonstrated the versatility of GPUs in different machine learning tasks.

Question 65

What was Sutskever's view on the limitations of support vector machines (SVMs)?

Answer 65

The ceiling for SVMs is low compared to neural networks ## Footnote He believed neural networks had a higher potential if trained correctly.

Question 66

What was the ImageNet challenge introduced in 2010?

Answer 66

Train a system to categorize 1.2 million images into 1,000 categories ## Footnote It aimed to advance the field of computer vision.

Question 67

What was the top-5 error rate achieved by AlexNet in the ImageNet competition?

Answer 67

17 percent ## Footnote This was significantly lower than the previous winners' rates.

Question 68

What activation function did AlexNet use for its neurons?

Answer 68

Rectified Linear Unit (ReLU) ## Footnote This was a departure from the traditional sigmoid function.

Question 69

What impact did deep neural networks have on various fields?

Answer 69

Revolutionized computer vision, natural language processing, machine translation, and more ## Footnote Their applications are vast and continually growing.

Question 70

What did Mikhail Belkin compare the current state of machine learning to in physics?

Answer 70

The emergence of quantum mechanics ## Footnote He suggested that deep neural networks are leading to new theories in machine learning.