Machine Translation and Encoder-Decoder Models

Question 1

What is machine translation?

Answer 1

It is the use of computers to translate one language to another

Question 2

What machine translation models exist?

Answer 2

Statistical phrase alignment models, Encoder-Decoder models and Transformer models

Question 3

What type of task is machine translation?

Answer 3

It is a sequence to sequence task (seq2seq)

Question 4

What is the input, output and their lengths for a seq2seq task?

Answer 4

The input X is a sequence of words, the output Y is a sequence of words, but the length of X may not necessarily equal the length of Y

Question 5

Besides machine translation, what are some other seq2seq tasks?

Answer 5

Question → Answer

Sentence → Clause

Document → Abstract

Question 6

What do universal aspects mean in regards to the human language?

Answer 6

These are aspects that are true, or statistically mostly true for all languages

Question 7

What are some examples of universal aspects in the human language?

Answer 7

Nouns/Verbs, Greetings, Politeness/Rude

Question 8

What are translation divergences?

Answer 8

These are areas where languages differ

Question 9

What are some examples of translation divergences?

Answer 9

Idiosyncrasies and lexical differences

Systematic differences

Question 10

What is the study of translation divergences called?

Answer 10

Linguistic Typology

Question 11

What is Word Order Typology?

Answer 11

It is a way of ordering words in different ways, these can include:

• Subject-Verb-Object (SVO)
• Subject-Object-Verb (SOV)
• Verb-Subject-Object (VSO)

Question 12

What is the Encoder-Decoder model?

Answer 12

For an input sequence, we have an encoder, that encodes the input to a context vector, which is then sent to a decoder that generates the output sequence.

Question 13

What can an encoder be?

Answer 13

LSTM, GRU, CNN, Transformers

Question 14

What is a context vector?

Answer 14

It is the last hidden layer of the encoder, which is used as the input to the decoder

Question 15

What does a language model try to do?

Answer 15

Predict the next word in a sequence Y based on the previous word

Question 16

How is a translation model different to a language model?

Answer 16

It predicts the next word in the sequence Y based on the previous target word AND the full source sequence X

Question 17

Explain how the encoder-decoder model shown in the image works

Answer 17

We have a single hidden layer that takes as an input the embeddings of the source text, we then have a separator and the predicted words based on its training. The predicted words are used in the prediction of the next word until the end is reached. The key is that the final hidden layer of the last input word is fed into the decoder which predicts the target words

Question 18

By using the hidden layer at the end of the sentence in the machine translation model, what is avoided?

Answer 18

It avoids the word typology problems as we have full knowledge of where the sentence starts and ends, meaning that before we start to translate the first word, we know what the end word is

Question 19

How is the encoder trained in machine translation models?

Answer 19

The input words are embedded using an embedding layer and are fed in one at a time to the encoder until the full input has been seen.

Question 20

Which state is the final hidden layer of the encoder fed into the decoder?

Answer 20

It is fed into every single state of the decoder

Question 21

What are the inputs at each step in the decoder?

Answer 21

The context vector c (final hidden state of encoder), the previous output, y_t-1 and the previous hidden state of the decoder h_t-1

Question 22

What is the typical loss function for a machine translation model?

Answer 22

It is a cross entropy loss function

Question 23

What is used during training of machine translation but not inference?

Answer 23

Teacher forcing, as we want to ensure we train on exact translations of the words

Question 24

What is the total loss per sentence in machine translation?

Answer 24

It is the averages loss across all target words

Question 25

Why is a separator token added to the start of the target sequence Y?

Answer 25

It is because the decoder needs a previous word embedding to compute a prediction, as nothing has been predicted, we need an initial input entry for the first word

Question 26

Given that the weights of previous tokens decay as the sequence is processed, how is access to previous hidden states without diminishing the weights achieved?

Answer 26

It is achieved using attention

Question 27

What can be used instead of the static context vector that has been shown to be better?

Answer 27

An attention vector

Question 28

What are some types of attention layers?

Answer 28

Dot-product attention

Additive attention

Self-attention

Question 29

What does the image show?

Answer 29

It shows how the context vector is replaced with the attention vector. Each of the hidden states are multiplied by some attention weight, and then a weighted sum is used to give us the context vector

Question 30

What problem can using an attention vector over the last encoder layer overcome?

Answer 30

In some languages, if the start/end of the sentence was more important, then some models would reverse the sentence to compensate for that. Attention avoids this through learning which parts to focus on

Question 31

What is a problem of using the argmax function with machine translation?

Answer 31

There is no hindsight to allow us to revisit choices at previous time steps if we have got something wrong.

Question 32

What is greedy decoding?

Answer 32

It is where argmax is used such that we cannot revisit previous choices - we are stuck with the choice we made

Question 33

What does Beam Search allow for?

Answer 33

It allows for you to revisit previous options if they become better than the current option that is currently being pursued

Question 34

How does a beam search decoder work?

Answer 34

It keeps a memory of the k-best sequence options (hypotheses) at any decoding step

Question 35

What is the beam width?

Answer 35

It is the memory size k

Question 36

What happens at each step in a beam search?

Answer 36

All k hypotheses are extended by V predicted tokens, which is the possible tokens.

The best k sequences from k x V hypotheses are then selected for memory

Question 37

Explain what the image below shows, using a beam width of 2.

Answer 37

We start with the start of the sentence. We first take the most probable two steps, so we have two sequences ‘start arrived’ and ‘start the’. At the next step, we have 4 possible options that we can choose. By using logs, we can add together the probabilities of each word in the sequence. In doing so, we can see that the two next most probable sequences from where we currently sit are ‘start the green’ and ‘start the witch’. We repeat this again, and see that the next most probable sequences are ‘start the green witch’ and ‘start the witch arrived’. This is repeated until we get to the end.

Question 38

What is the typical beam width in actual systems, and what is the problem of going above this size?

Answer 38

The typical size is 4 to 10, bigger than 10 can take a long time to decode which can slow the cycles.

Question 39

What type of vocabulary is used in a seq2seq model?

Answer 39

A fixed vocabulary (e.g. 50k limited by GPU memory) although BPE or WordPiece can also be used

Question 40

What decoder should be used to get fast results, and what should be used to get the best results?

Answer 40

Greedy decoder for fast, Beam decoder for best

Question 41

What are the training data options for machine translation?

Answer 41

Parallel Corpus - A corpus of two languages containing sentences that are aligned

Monolingual Corpus - These are stacks of posts in one language and stacks of posts in another, although there is no connection between the two. They are very large but require interesting techniques

Question 42

What is backtranslation?

Answer 42

Train a model using a parallel corpus, apply the model to the massive monolingual corpus and use this to train the model as it is sentence aligned and compare

Question 43

How can Machine Translation systems be evaluated?

Answer 43

Human Assessment, BLEU metric, Precision, Recall, NIST, TER, METEOR