Sequence Processing, Transformers and Attention

Question 1

What does LSTM stand for?

Answer 1

Long Short Term Memory

Question 2

Why is long distant information critical to many language apps?

Answer 2

Words at the start of the sentence can often have a large impact on the end of the sentence

Question 3

What is an issue of using RNNs?

Answer 3

The information value from the start of the sentence can degrade as we move down the sentence and becomes less accessible

Question 4

What is the vanishing gradients problem?

Answer 4

The vanishing gradients problem is where gradients get smaller and smaller as Gradient Descent progresses to deeper stacked layers, eventually ‘vanishing’. The connection weights are virtually unchanged and therefore training loss does not converge

Question 5

What is the exploding gradients problem?

Answer 5

It is the opposite of the vanishing gradients problem - as Gradient Descent progresses, the gradients get larger and larger causing the training loss to not converge.

Question 6

What does the image show?

Answer 6

This is an LSTM cell, an LSTM layer consists of many of these cells

Question 7

Explain how an LSTM cell, similar to the one shown in the image, works.

Answer 7

The last hidden layer, h_t-1, is still provided as an input alongside the current input x_t. An additional memory unit, which is c_t-1, which is a form of secondary memory that can record longer term things. The LSTM cell can be trained to learn what to remember and what to forget, which is where the gates are used. Each gate is trained with an activation function in order to compute their values. Given the inputs, they go through the gates and pass through the activation function to generate output values, which in turn generate the output vector y_t, the next long term state c_t and the next short term state h_t. To do the training, each gate has a fully connected dense layer which is used to train and update the weights.

Question 8

What is a GRU?

Answer 8

It is a Gated Recurrent Unit

Question 9

What does a GRU do?

Answer 9

It merges both long and short term state vectors

Question 10

Are GRUs simpler or more complex than an LSTM?

Answer 10

They are simpler and more efficient to compute while still producing similar results.

Question 11

In simple terms, explain the difference between the 4 concepts shown in the image.

Answer 11

The MLP simply uses the input to produce the output

The RNN will aggregate the last hidden layer as well as the input X

The LSTM cell will aggregate the last hidden layer, has an input X, and has long term memory while learning what to remember

The GRU simply merges the long term memory with the short term memory and learns what to remember and forget

Question 12

What does the transformer architecture show?

Answer 12

Using the concept of self-attention, the LSTM and GRUs could be outperformed

Question 13

What are the problems of GRUs and RNNs that transformers solve?

Answer 13

While they were designed to help with longer distance sequences, they still could not cope with very long distant sequences

Question 14

What is the key concept of the transformer architecture?

Answer 14

It uses an attention layer that does not worry about the length of the sequence as it has access to all potential inputs of the sequence, learning which ones to attend to.

Question 15

Explain the architecture of the image showing the Transformer model.

Answer 15

It is an encoder-decoder model. We have stacks of multi-head attention layers for the encoder, and stacks of multi-head attention layers for the decoder, of which we have N times of them. The encoder feeds into the decoder and we have a linear layer with softmax to give an output to the problem.

Question 16

Why is a positional encoding used in the transformer model?

Answer 16

It is because the model takes all the words as one big input vector, so it does not have access to the sequential positions of each of the words. Therefore, positional encoding is added to provide some knowledge of the positions

Question 17

What are attention layers?

Answer 17

They compare items to other items to reveal their relevance in the current context

Question 18

How does the self-attention layer map input sequences?

Answer 18

It maps input sequences to the output sequences of the same length, we have contributions from all the other inputs in the sequence up until that point

Question 19

What is an advantage of using self-attention layers?

Answer 19

The computation for each output is computationally independent, meaning that it can be parallelized and efficient to compute

Question 20

In self-attention, how do we compare items?

Answer 20

All inputs are embedding vectors for each of the words which are then processed in pairs. All previous inputs are compared to the current input using the dot product function

Question 21

What equations are required to compare the items with self-attention?

Answer 21

1. For sequence x₁..x₃ we need to compute score (x₃,x₁), score (x₃,x₂) and score (x₃,x₃)
2. Then we normalise each score using softmax to create a vector of attention weights called alpha
3. Finally, we compute the output by summing the inputs in the sequence weighted by alpha

Question 22

How can we provide trainable weights for the attention function?

Answer 22

We use several weight matrices for this

Question 23

Why does a transformer use embeddings?

Answer 23

They make the approach differentiable, meaning it can be used with gradient descent and therefore it fits well within our deep learning stack - if it was not then we would not be able to compute loss and perform the training cycle

Question 24

In self-attention, what does the matrix Q allow us to do?

Answer 24

It allows us to map the input embedding x to the query embedding q

Question 25

In self-attention, what does the matrix K allow us to do?

Answer 25

It allows us to map the input embedding x to be mapped to the key embedding k

Question 26

In self-attention, what does the matrix V allow us to do?

Answer 26

It allows the input embedding x to mapped to a potential value embedding v

Question 27

How is the score for an input pair computed using self-attention?

Answer 27

We take the dot product of the query embedding for the current word and the key embedding of the previous word, divided by the square-root of key dimension, ensuring the weights are appropriate

Question 28

When applying the self-attention, what do we do?

Answer 28

We compute the similarity of the queries and the keys using the softmax function

Question 29

What do we do with future characters when we are training?

Answer 29

We mask the values after the current position as using information after the current position to train would be cheating

Question 30

How are positional embeddings learnt?

Answer 30

They are learnt either during training (e.g. additional position input alongside the words) or they can be generated using a positional encoding function (e.g. cosine/sine function)

Question 31

What is important about the dimensions of the positional embeddings?

Answer 31

They are the same as the dimensions of the word embeddings so they can be summed together to give a position-aware input embedding

Question 32

What is a self-attention layer called?

Answer 32

A head

Question 33

Why do we want N heads?

Answer 33

It is the same reason as to why we use stacked RNNs, using multiple of them can capture different patters, so more layers can lead to more complicated patterns being captured around the language used

Question 34

What happens to the output of all the heads used in a transformer?

Answer 34

They are concatenated

Question 35

How does a transformer work in regards to a text completion task?

Answer 35

The input is a sequence of words, the output is a prediction of words to complete the sequence. The transformer takes the input text as an embedding, a number of transformer blocks which are N stacks of multihead self-attention layers, which is passed to a softmax function to make a prediction for the most probable output word

Question 36

What is used during training with a transformer for text completion but not in testing/actual use?

Answer 36

During training, teacher forcing is likely used to ensure it is correct, but in the inference mode, the predicted word is used to predict the following word, and so.