Terminology

Question 1

What is an epoch?

Answer 1

One pass of all training samples

Question 2

What is a batch and batchsize?

Answer 2

The training samples are collected in smaller batches. The batchsize is the number of samples in one batch.

Question 3

What does the number of steps/iterations mean?

Answer 3

The number of batches it takes to go though an epoch.

Question 4

What is a checkpoint?

Answer 4

Different model versions saved throughout training

Question 5

What is BERT?

Answer 5

Bidirectional Encoder Representations from Transformers.
BERT is an encoder-only model.
A pre-trained encoder stack that creates dynamic or contextualised word embeddings which can be fine-tuned for transfer learning.

Question 6

What is GPT2?

Answer 6

GPT2 is a decoder-only model

Question 7

What is warm-starting?

Answer 7

Composing an encoder-decoder model of pre-trained stand-alone model checkpoints is defined as warm-starting the encoder-decoder model

Question 8

What is Fine-tuning?

Answer 8

The task-specific training of a model that has been initialized with the weights of a pre-trained language model

Question 9

What defines a Pre-trained model?

Answer 9

It has been trained on unlabeled text data, i.e. in a task-agnostic, unsupervised fashion, and it processes a sequence of input words into a context-dependent embedding.

Question 10

What is self-attention?

Answer 10

Self-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the same sequence.

Overall, the self-attention mechanism allows the inputs to interact with each other (“self”). A self-attention module works by comparing every word in the sentence to every other word in the sentence, including itself, and reweighing the word embeddings of each word to include contextual relevance.

It adds contextual information to the words in the sentence. e.g. to provide the best representation of words with different meanings.

Question 11

What is an RNN?

Answer 11

RNNs are specialized for processing sequential data where the order is meaningful. e.g. language.
Recurrent neural networks are unidirectional neural networks meaning they only work sequentially and introduce a hidden state to which is passed along the chain. E.g. saying the alphabet is easy going forward but not backwards because it is learned as a sequence.

Question 12

What is a CNN?

Answer 12

CNNs are especially good for image data.

A convolution neural network can process input in parallel but has finite memory.

Question 13

What stands T5 for?

Answer 13

T5 stands for “Text-to-Text Transfer Transformer”

Question 14

What is the difference/similarity between BERT and mT5?

Answer 14

Both are based on transformers. BERT in itself is just an encoder so not default a text-to-text model as T5.

Question 15

What is a Transformer architecture?

Answer 15

An attention-only sequence-to-sequence architecture introdcued by Vaswani et al. Advantages: Input sequence can be passed in parallel. A model with multiple attention heads processed in parallel with its own set of weight matrices allows the model to jointly attend to different attention focuses in the information from different representational aspects in each sub-layer. Uses positional encoding.

Question 16

How is a GPU an advantage for the summariser?

Answer 16

For the mT5 summariser, it includes neural networks, which utilizes matrix multiplication (MM). MM can be processing in parallel using block MM. Cuda takes care of this allocation of memory and hyperparameters concerning this.

Question 17

What is an LSTM?

Answer 17

Long short term memory.
Allows past information to skip a lot of processing, so memory can be retained for longer sequences.
Solves some of the problems with RNNs but are slow and complex since they work serially and therefore not good for GPUs.

(Transfer learning not very good for LSTM. You need a new labeled data set specific to your task everytime.)

Question 18

What is the encoder-decoder attention layer?

Answer 18

Attends to specific parts of the input in relation to the output.

Question 19

What is the Text Rank method?

Answer 19

Using each sentence as a node and then and edge between two nodes if they have a similarity above a certain threshold. The more edges between nodes the more important is the node.

Question 20

What is word embedding?

Answer 20

Word embedding algorithms turn words into vector spaces that capture a lot of the meaning/semantic information of the words (e.g. king - man + woman = queen).
E.g. GloVe or continuous neural net like this one (in our project).

Question 21

What is attention?

Answer 21

Attention allows the model to focus on the relevant parts of the input sequence as needed.

Introduced since the context vector (the last hidden state of the RNN encoder passed to the decoder) was problematic for longer sentences.
Now all hidden states are passed to the decoder model. The decoder scores the hidden states i.e. which are more relevant.

Question 22

What word embeddings does the mT5 use?

Answer 22

Contextualised embeddings (see Ross’) slides on “die”

Question 23

What is an encoder?

Answer 23

An encoder processes the input sequence and compresses the information into a context vector also called sentence embedding.
A summary of the meaning in a sequence.

Question 24

What is a decoder?

Answer 24

A decoder is initialized with the context vector to emit the transformed output.

Question 25

what is bag-of-words?

Answer 25

One-hot encoding words. The downfall is that it is simple and does not capture the order of words or grammar. Ant is not computationally efficient since the vector needs to be all words combined / longest sequence.

Question 26

How does the mT5/transformer represent words?

Answer 26

A defining feature of neural network language models is their representation of words as high dimensional real-valued vectors. As continuous space word representations. So-called contextualized
embeddings. Not static.

Question 27

Why non-linear weights?

Answer 27

Without non-linearities, deep neural networks can’t do anything more than a linear transform. With more layers, they can approximate more complex functions! Non-linearities allow for better decision boundaries in more complex scenarios.

Question 28

What is the distributional hypothesis?

Answer 28

This link between similarity in how words are distributed and similarity in what they mean.

Question 29

What is static word embedding?

Answer 29

Word2vec embeddings are static embeddings, meaning that the method learns one fixed embedding for each word in the static embeddings vocabulary. Also Skip-gram is a static word embedding that trains a logistic regression classifier to compute the probability that two words are ‘likely to occur nearby in text’.

Question 30

What are contextualised embeddings?

Answer 30

Each word will be represented by a

different vector each time it appears in a different context.

Question 31

What is the vanishing gradient problem?

Answer 31

Involves the weights in earlier layers of the network. SGD uses the gradient value computed to update the weights. If gradients are small aka vanishing, the weights won’t change. The weights become stuck. (Same with exploding gradients.)

Represents a general problem with unstable gradients. Specifically, gradient-based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases such as long-distance dependencies in language.

Question 32

What is an encoder-decoder model architecture?

Answer 32

Encoding means converting data into a required format. Put together with a decoder meaning converting a coded message into intelligible language

Question 33

What is a T5 model exactly?

Answer 33

T5 uses a unified “text-to-text” format which is natural for generative tasks but not as natural for classification tasks where T5 will have to output the string label and not an index. T5 is trained similarly to BERT on masked language tasks, but sometimes replace multiple consecutive tokens with a single Mask keyword → since the model needs to output text it is designed to target a sequence.