DL-05b - Convolutional neural network - Architectures

Question 1

DL-05b - Convolutional neural network - Architectures

Who is behind LeNet?

Answer 1

Yann LeCun

Question 2

DL-05b - Convolutional neural network - Architectures

When did Yann LeCun publish LeNet?

Answer 2

1998

Question 3

DL-05b - Convolutional neural network - Architectures

What was LeNet used for?

Answer 3

Reading zip codes/digits.

Question 4

DL-05b - Convolutional neural network - Architectures

Who created AlexNet?

Answer 4

Alex Krishevsky

Question 5

DL-05b - Convolutional neural network - Architectures

When did Alex Krishevsky publish AlexNet?

Answer 5

2012

Question 6

DL-05b - Convolutional neural network - Architectures

What network was published in 1998?

Answer 6

LeNet

Question 7

DL-05b - Convolutional neural network - Architectures

What network was published in 2012?

Answer 7

AlexNet

Question 8

DL-05b - Convolutional neural network - Architectures

What challenge did AlexNet attempt to solve?

Answer 8

ImageNet,

Question 9

DL-05b - Convolutional neural network - Architectures

How many different labels does ImageNet have?

Answer 9

1,000

Question 10

DL-05b - Convolutional neural network - Architectures

What’s the architecture of AlexNet?

Answer 10

5 conv layers, 3 FC layers, overlapping max pooling

Question 11

DL-05b - Convolutional neural network - Architectures

How many parameters did AlexNet have?

Answer 11

~62 million

Question 12

DL-05b - Convolutional neural network - Architectures

When was VGGNet published?

Answer 12

2014

Question 13

DL-05b - Convolutional neural network - Architectures

Who published VGGNet?

Answer 13

Simonyan and Zisserman

Question 14

DL-05b - Convolutional neural network - Architectures

How many layers did VGGNet have?

Answer 14

16, hence it’s often called VGG16.

Question 15

DL-05b - Convolutional neural network - Architectures

What was the key innovation of VGGNet?

Answer 15

Depth of network is key to performance.

Question 16

DL-05b - Convolutional neural network - Architectures

What were some drawbacks of VGGNet relative to its contemporaries? (2)

Answer 16

• more expensive to evaluate
• requires large memory and computation power.

Question 17

DL-05b - Convolutional neural network - Architectures

Whwne was GoogLeNet published?

Answer 17

2014

Question 18

DL-05b - Convolutional neural network - Architectures

Who published GoogLeNet?

Answer 18

Szegedy et al.

Question 19

DL-05b - Convolutional neural network - Architectures

How deep was GoogLeNet?

Answer 19

22 layers

Question 20

DL-05b - Convolutional neural network - Architectures

What key innovations did GoogLeNet have? (3)

Answer 20

• Reduced parameters with better performance
• Inception modules
• Average pooling instead of FC layers

Question 21

DL-05b - Convolutional neural network - Architectures

What is the inception module?

Answer 21

Block that splits into multiple CNN blocks with different kernel sizes.

Question 22

DL-05b - Convolutional neural network - Architectures

In inception modules, what do concatenation layers do?

Answer 22

Combines all the outputs and feature maps from the conv filters into one single output of the Inception module.

Question 23

DL-05b - Convolutional neural network - Architectures

In inception modules, what solves the problem of high computation cost?

Answer 23

1x1 conv blocks to reduce number of input channels.

(192 -> 1x1-> 16) -> (16 -> 5x5 -> 32)

(See image)

Question 24

DL-05b - Convolutional neural network - Architectures

Describe the full inception module.

Answer 24

(See image)

Question 25

# DL-05b - Convolutional neural network - Architectures What is depicted? (See image)

Answer 25

A full inception module.

Question 26

# DL-05b - Convolutional neural network - Architectures What is depicted? (See image)

Answer 26

A naive inception module.

Question 27

# DL-05b - Convolutional neural network - Architectures What are the benefits of using inception modules? (3)

Answer 27

- Extracts features at varying scales using varying filter sizes - minimizes computation load - utilizes 1x1 conv filters for better cross-channel pattern learning

Question 28

# DL-05b - Convolutional neural network - Architectures When was Inception 4 released?

Answer 28

2016

Question 29

# DL-05b - Convolutional neural network - Architectures What tricks are introduced to make Inception better later on? (2)

Answer 29

Improvements to speed through: - (5x5) factored into 2 x (3x3) convs - (nxn) factored into (1xn) and (nx1).

Question 30

# DL-05b - Convolutional neural network - Architectures When was ResNet published?

Answer 30

2015

Question 31

# DL-05b - Convolutional neural network - Architectures Who was behind the ResNet paper?

Answer 31

He et al.

Question 32

# DL-05b - Convolutional neural network - Architectures Which company published ResNet?

Answer 32

Microsoft.

Question 33

# DL-05b - Convolutional neural network - Architectures How many layers did the original ResNet model have?

Answer 33

152

Question 34

# DL-05b - Convolutional neural network - Architectures What idea did ResNet introduce?

Answer 34

Residual learning

Question 35

# DL-05b - Convolutional neural network - Architectures What block is this? (See image)

Answer 35

A residual block

Question 36

# DL-05b - Convolutional neural network - Architectures What intuition can we use for ResNets?

Answer 36

The residual blocks makes the layers comparable to ensembles of different models.

Question 37

# DL-05b - Convolutional neural network - Architectures What are the "two modes" for how layers are trained in ResNets?

Answer 37

Layers are either trained or training is skipped for layers using skip connections.

Question 38

# DL-05b - Convolutional neural network - Architectures When were Inception-ResNets published?

Answer 38

2017

Question 39

# DL-05b - Convolutional neural network - Architectures Who were behind Inception-Resnets?

Answer 39

Szegedy et al.

Question 40

# DL-05b - Convolutional neural network - Architectures What is SENet short for?

Answer 40

Squeeze and Excitation Networks

Question 41

# DL-05b - Convolutional neural network - Architectures Who were behind SENet?

Answer 41

Hu et al.

Question 42

# DL-05b - Convolutional neural network - Architectures When was SENet published?

Answer 42

2017

Question 43

# DL-05b - Convolutional neural network - Architectures What does a squeeze-and-excitation block attempt to do?

Answer 43

Enables the network to perform dynamic channel-wise feature recalibration.

Question 44

# DL-05b - Convolutional neural network - Architectures What is the name of the block that attempts to perform dynamic channel-wise feature recalibration?

Answer 44

Squeeze-and-excitation block

Question 45

# DL-05b - Convolutional neural network - Architectures In a squeeze-and-excitation block, what is the purpose of squeezing?

Answer 45

squeeze global spatial information into a channel descriptor.

Question 46

# DL-05b - Convolutional neural network - Architectures In a squeeze-and-excitation block, what is the purpose of excitation?

Answer 46

Excitation operation is used to fully capture channel-wise dependencies.

Question 47

# DL-05b - Convolutional neural network - Architectures Squeeze-and-excitation block looks like?

Answer 47

(See image)

Question 48

# DL-05b - Convolutional neural network - Architectures What is the technique used in EfficientNet called?

Answer 48

Compound scaling

Question 49

# DL-05b - Convolutional neural network - Architectures Which network introduced Compound scaling?

Answer 49

EfficientNet

Question 50

# DL-05b - Convolutional neural network - Architectures What is Compound scaling?

Answer 50

An equation for setting the width, depth and image resolution of a network.

Question 51

# DL-05b - Convolutional neural network - Architectures What is the equation for compound scaling?

Answer 51

(See image)

Question 52

# DL-05b - Convolutional neural network - Architectures What is MBConv short for?

Answer 52

Mobile inverted bottleneck

Question 53

# DL-05b - Convolutional neural network - Architectures What is the shortened name of Mobile inverted bottleneck?

Answer 53

MBConv

Question 54

# DL-05b - Convolutional neural network - Architectures What is the main idea behind MBConv?

Answer 54

They use an inverted structure for efficiency reasons.

Question 55

# DL-05b - Convolutional neural network - Architectures What does MBConv blocks do? (3 steps)

Answer 55

- Increases number of channels with 1x1 conv - Depth-wise convolutions - Downsample channels to initial value using 1x1 conv

Question 56

# DL-05b - Convolutional neural network - Architectures Who is behind EfficientNet V2?

Answer 56

Tan and Le.

Question 57

# DL-05b - Convolutional neural network - Architectures When was EfficientNet V2 published?

Answer 57

2021

Question 58

# DL-05b - Convolutional neural network - Architectures What does EfficientNet V2 use? (3) (BLR)

Answer 58

- Combines MBConv and Fused-MBConv blocks - Uses progressive learning (changes image size dynamically) - Uses adaptive regularization

Question 59

# DL-05b - Convolutional neural network - Architectures How does ResNet tackle the error propagation challenge?

Answer 59

Through skip connections.

Question 60

# DL-05b - Convolutional neural network - Architectures What makes error propagation efficient in a ResNet architecture?

Answer 60

Residual connections allow error to bypass layers, ensuring efficient error propagation and training.

Question 61

# DL-05b - Convolutional neural network - Architectures What architecture is this? (See image)

Answer 61

MBConv

Question 62

# DL-05b - Convolutional neural network - Architectures Label the masked parts.

Answer 62

(See image)

Question 63

# DL-05b - Convolutional neural network - Architectures Describe the full architecture of MBConv.

Answer 63

(See image)

Question 64

# DL-05b - Convolutional neural network - Architectures What architecture is this? (See image)

Answer 64

Fused MBConv

Question 65

# DL-05b - Convolutional neural network - Architectures Label the masked parts.

Answer 65

(See image)

Question 66

# DL-05b - Convolutional neural network - Architectures Describe what parts a fused-MBConv consists of.

Answer 66

(See image)