DL-08 - Transformers

Question 1

DL-08 - Transformers

What are some problems with RNNs/LSTMs? (4)

Answer 1

• Difficult to train.
• Very long gradient paths.
• Transfer learning never really works.
• Recurrence is against the principle of parallel computation.

Question 2

DL-08 - Transformers

What is the name of the paper where transformers were introduced?

Answer 2

Attention is All You Need

Question 3

DL-08 - Transformers

When was the transformers paper (Attention is All You Need) published?

Answer 3

2017

Question 4

DL-08 - Transformers

Who were the authors of the transformers paper (Attention is All You Need)

Answer 4

Vaswani et al.

Question 5

DL-08 - Transformers

What do transformers use instead of recurrence? (2)

Answer 5

• Context windows (input more data at the same time)
• self-attention

Question 6

DL-08 - Transformers

In what areas are transformers currently very good? (2)

Answer 6

• NLP
• Computer vision

Question 7

DL-08 - Transformers

What does a transformer do? (2)

Answer 7

• Encodes an input into a single vector
• Decodes the vector back into output

Question 8

DL-08 - Transformers

Does transformers use recursion?

Answer 8

No, they avoid it.

Question 9

DL-08 - Transformers

Why can encoders be so fast?

Answer 9

No recursion -> parallel computation.

Question 10

DL-08 - Transformers

What are the main characteristics of transformers? (3)

Answer 10

• non-sequential
• self-attention
• positional encoding

Question 11

DL-08 - Transformers

Describe what is meant when we say transformers are non-sequential.

Answer 11

Sentences are processed as a whole, rather than word by word.

Question 12

DL-08 - Transformers

“Sentences are processed as a whole, rather than word by word.”
What is this property called?

Answer 12

Non-sequential.

Question 13

DL-08 - Transformers

Describe self-attention.

Answer 13

A new unit used to compute similarity scores between words in a sentence.

Question 14

DL-08 - Transformers

Describe positional encoding.

Answer 14

Encodes information related to a position of a token in a sentence.

Question 15

DL-08 - Transformers

“Encodes information related to a position of a token in a sentence.”
What is this called?

Answer 15

positional encoding

Question 16

DL-08 - Transformers

“A new unit used to compute similarity scores between words in a sentence.”
What is this called?

Answer 16

Self-atttention.

Question 17

DL-08 - Transformers

What is the method transformers use to understand relevant words while processing a current word?

Answer 17

Self-atttention.

Question 18

DL-08 - Transformers

Why don’t transformers suffer from short-term memory?

Answer 18

Because they use self-attention mechanisms, allowing them to take the entire input sequence into account simultaneously.

Question 19

DL-08 - Transformers

What parts does “encoder embedding” consist of?

Answer 19

• Word/input embedding
• Positional embedding

Question 20

DL-08 - Transformers

What does positional embedding do?

Answer 20

It injects positional information (distance between different words) into the input embeddings.

Question 21

DL-08 - Transformers

What functions does the “Attention is all you need” paper use for positional encoding?

Answer 21

Sin/cos

Question 22

DL-08 - Transformers

In the image, what is
- d_model
- i
- pos

(See image)

Answer 22

• d_model: Embedding size
• i: Depends on the position in the embedding dimension
• pos: Position index in the incoming sequence.

Question 23

DL-08 - Transformers

How is positional information added to the embeddings?

Answer 23

They’re added element-wise (Addition).

Question 24

DL-08 - Transformers

What are the sub-modules of the encoder? (2)

Answer 24

• Multi-headed attention
• Fully connected feed forward network

Question 25

# DL-08 - Transformers What do each of the sub-modules have (both attention head and FC module)? (2)

Answer 25

- Residual connections - Normalization layer

Question 26

# DL-08 - Transformers What are the vectors in self-attention called? (3)

Answer 26

- Query - Key - Value

Question 27

# DL-08 - Transformers How are the query, key and value vectors created?

Answer 27

Separate weight matrix for each. Each vector is simply a multiplication with the incoming embedding vector. (See image)

Question 28

# DL-08 - Transformers How is "score" calculated in self-attention? @CHECK ME

Answer 28

By taking the dot product of the Q and K vectors.

Question 29

# DL-08 - Transformers What do you get when you take the dot product of the Q and K vectors? @CHECK_ME

Answer 29

The "score".

Question 30

# DL-08 - Transformers How do you ensure stable gradients?

Answer 30

Scale down the "scores".

Question 31

# DL-08 - Transformers What is the formula for scaling the scores?

Answer 31

(See image)

Question 32

# DL-08 - Transformers How do you normalize the scores?

Answer 32

Use softmax to produce attention weights with probability between 0-1.

Question 33

# DL-08 - Transformers How do you calculate the final attention weights?

Answer 33

Use softmax normalize the scores, to produce attention weights with probability between 0-1.

Question 34

# DL-08 - Transformers How do you get the output vector of a self-attention unit?

Answer 34

Calculate the self-attention score/weights, then multiply it by the value vector.

Question 35

# DL-08 - Transformers How do you write the self-attention block as a single matrix operation?

Answer 35

(See image)

Question 36

# DL-08 - Transformers What is depicted in the image? (See image)

Answer 36

Most of the self-attention block as a matrix operation.

Question 37

# DL-08 - Transformers What is Multi-headed attention?

Answer 37

A block that uses N different self-attentions (called heads) with different Q, K, V to produce outputs Z_1, Z_2, Z_3, ..., Z_n.

Question 38

# DL-08 - Transformers What is it called when you use multiple self-attention blocks in the same layer?

Answer 38

Multi-headed attention

Question 39

# DL-08 - Transformers What is a self-attention block called?

Answer 39

A head.

Question 40

# DL-08 - Transformers What is a head?

Answer 40

One self-attention block.

Question 41

# DL-08 - Transformers What does multi-headed attention do for the layer?

Answer 41

It allows the layer to have multiple representation subspaces.

Question 42

# DL-08 - Transformers What is done to the outputs of the individual self-attention blocks, to make them outputs of a multi-headed attention block?

Answer 42

They're concatenated into single matrix and multiplied with a weight matrix W^O.

Question 43

# DL-08 - Transformers What is in the image? (See image)

Answer 43

A transformer encoder block.

Question 44

# DL-08 - Transformers Label the masked parts of the image. (See image)

Answer 44

(See image)

Question 45

# DL-08 - Transformers What is in the image? (See image)

Answer 45

A transformer decoder block

Question 46

# DL-08 - Transformers Label the masked parts of the image. (See image)

Answer 46

(See image)

Question 47

# DL-08 - Transformers What happens to the outputs of a transformer decoder block?

Answer 47

(See image)

Question 48

# DL-08 - Transformers What sub-layers does the decoder have? (3)

Answer 48

- 2 multi-headed attention layers, - a feed-forward layer, - residual connections and normalization layers after each sub-layer

Question 49

# DL-08 - Transformers What are decoder embedding comprised of? (2)

Answer 49

- Output word embedding - Positional embedding

Question 50

# DL-08 - Transformers What is fed into the first multi-head attention layer in a Transformer decoder?

Answer 50

The output of the Transformer decoder embedding.

Question 51

# DL-08 - Transformers In the transformer's decoder, how is the first attention head different than the encoder's attention head?

Answer 51

It uses a look-ahead mask.

Question 52

# DL-08 - Transformers In sequence models, what is the purpose of a look-ahead mask used in a decoder with multi-head attention?

Answer 52

To prevent the decoder from conditioning to future tokens.

Question 53

# DL-08 - Transformers How do you create a look-ahead mask?

Answer 53

(See image)

Question 54

# DL-08 - Transformers Where is the mask applied in a decoder?

Answer 54

(See image)

Question 55

# DL-08 - Transformers What are the input to the decoder's 2nd attention head? (2)

Answer 55

- Query and key from encoder - Value from 1st attention head.

Question 56

# DL-08 - Transformers What is another way to think of the 2nd attention head in the decoder?

Answer 56

Encoder-decoder attention

Question 57

# DL-08 - Transformers What happens to the output of the decoder block?

Answer 57

It's sent through a linear classifier, then a softmax activation. (See image)

Question 58

# DL-08 - Transformers How do we interpret the output of a transformer?

Answer 58

It's a probability distribution over the words in your vocabulary. (We try to predict the next word.)

Question 59

# DL-08 - Transformers What is a stacked encoder/decoder?

Answer 59

Adding multiple layers of encoders/decoders to improve performance. (See image)

Question 60

# DL-08 - Transformers What are some popular transformers mentioned in the paper? (5)

Answer 60

- BERT - OpenAI's GPT family - Google Bard - XLNet - T5

Question 61

# DL-08 - Transformers What is BERT short for?

Answer 61

Bidirectional Encoder Representations from Transformers

Question 62

# DL-08 - Transformers What is GPT short for?

Answer 62

Generative Pretrained Transformer

Question 63

# DL-08 - Transformers What is T5 short for? (TTTTT)

Answer 63

Text-To-Text Transfer Transformer

Question 64

# DL-08 - Transformers When was BERT released?

Answer 64

2018

Question 65

# DL-08 - Transformers When was the first GPT released?

Answer 65

2018

Question 66

# DL-08 - Transformers When was XLNet released?

Answer 66

2020

Question 67

# DL-08 - Transformers When was T5 released?

Answer 67

2020

Question 68

# DL-08 - Transformers What are the two novel techniques used by BERT? (2)

Answer 68

- Masked Language Model (MLM) - Next Sentence Prediction (NSP)

Question 69

# DL-08 - Transformers What is MLM short for?

Answer 69

Masked Language Model

Question 70

# DL-08 - Transformers What is NSP short for?

Answer 70

Next Sentence Prediction

Question 71

# DL-08 - Transformers What does BERT used to better determine context?

Answer 71

Bidirectional

Question 72

# DL-08 - Transformers What are some tasks where BERT is useful? (3)

Answer 72

- Classification - Fill in the blanks - Question answering

Question 73

# DL-08 - Transformers What variants of BERT mentioned in the lecture slides? (4)

Answer 73

- RoBERTa - ALBERT - StructBERT - DeBERTa

Question 74

# DL-08 - Transformers What's special about RoBERTa?

Answer 74

A Robustly Optimized BERT Pretraining Approach

Question 75

# DL-08 - Transformers What's special about ALBERT?

Answer 75

A Lite BERT for Self-supervised Learning of Language Representations

Question 76

# DL-08 - Transformers What's special about StructBERT?

Answer 76

Incorporating Language Structures into Pre-training for Deep Language Understanding

Question 77

# DL-08 - Transformers What objective was GPT trained with?

Answer 77

Predicting the next word in a sequence.

Question 78

# DL-08 - Transformers How are GPTs traiend?

Answer 78

Using RLHF (Reinforcement learning from human feedback)

Question 79

# DL-08 - Transformers What is RLHF short for?

Answer 79

Reinforcement learning from human feedback

Question 80

# DL-08 - Transformers How many layers does GPT-3 have?

Answer 80

96 layers

Question 81

# DL-08 - Transformers How many attention heads per layer does GPT-3 have?

Answer 81

96 attention heads

Question 82

# DL-08 - Transformers What is a vision transformer?

Answer 82

Using transformers for computer vision?

Question 83

# DL-08 - Transformers What is ViT short for?

Answer 83

Vision transformer

Question 84

# DL-08 - Transformers Who first published vision transformers for imagenet?

Answer 84

Dosovitskiy et al. from Google Brain

Question 85

# DL-08 - Transformers When were vision transformers first published?

Answer 85

2020

Question 86

# DL-08 - Transformers What is the architecture of vision transformers?

Answer 86

(See image)

Question 87

# DL-08 - Transformers How are images preprocessed for use in a vision transformer?

Answer 87

Data is split into small patches, e.g. 16x16.

Question 88

# DL-08 - Transformers Label the masked parts of the image.

Answer 88

(See image)

Question 89

# DL-08 - Transformers Label the masked parts of the image.

Answer 89

(See image)

Question 90

# DL-08 - Transformers Label the masked parts of the image.

Answer 90

(See image)

Question 91

# DL-08 - Transformers Label the masked parts of the image.

Answer 91

(See image)

Question 92

# DL-08 - Transformers Label the masked parts of the image.

Answer 92

(See image)

Question 93

# DL-08 - Transformers Label the masked parts of the image.

Answer 93

(See image)

Question 94

# DL-08 - Transformers What's depicted in the image?

Answer 94

A ViT (Vision transformer).