Introduction to Transformers for NLP

Question 1

RNN

Answer 1

Recurrant Neural Network

In an RNN, the information is repeated endlessly within a loop

Question 2

Bag of Words

Question 3

n-grams

Question 4

trigram

Answer 4

A trigram model keeps the context of the last two words to predict the next word in the sequence

Question 5

LSTM

Answer 5

Long Short-Term Memory

Question 6

GRU

Answer 6

Gated Reccurrent Unit

Question 7

Feed-forward Neural Network

Question 8

BP Mechanism

Answer 8

Back Propogation

Question 9

Gradient Decent

Question 10

T5 model

Question 11

seq2seq

Answer 11

sequence-to-sequence neural network

Question 12

multi-head attention

Answer 12

multiple modules of self-attention capturing different kinds of attentions.

Question 13

feed-forward

Question 14

masked multi-head attention

Question 15

linear

Question 16

softmax

Answer 16

The softmax function is a mathematical function that converts a vector of real numbers into a probability distribution, where each value is between 0 and 1 and all values sum to 1.

Question 17

input embeddings

Question 18

output embeddings

Question 19

tokenize

Question 20

vectorize

Question 21

positional encoding

Question 22

self-attention

Answer 22

self-attention allows us to associate each word in the input with other words in the same sentence

Question 23

query vector

Question 24

key vector

Question 25

value vector

Question 26

embedding vector

Question 27

residual connection

Answer 27

The original positional input embedding is then given the multi-headed attention output vector as an additional component

Question 28

decoder

Answer 28

1. Multi-headedattention layer 2. Add and norm layers 3. Feed-forward layer

Question 29

encoder

Question 30

BERT

Answer 30

Bidirectional Encoder Representations from Transformers

Question 31

BERT-Base

Answer 31

BERT-Base has a total of 110 million parameters, 12 attention heads, 768 hidden nodes, and 12 layers.

Question 32

BERT-Large

Answer 32

BERT-Large is characterized by having 24 layers, 1024 hidden nodes, 16 attention heads, and 340 million parameter values.

Question 33

Masked-LM

Answer 33

MLM

Question 34

NSP

Answer 34

Next Sentence Prediction

Question 35

CLS

Answer 35

Always consider the first token in a sequence to be a special classification token (also abbreviated CLS)

Question 36

SEP

Answer 36

The [SEP] token serves to demarcate the break between the two sentences.