05 - Deep Learning for Sequence Processing

Question 1

What is an encoder and a decoder?

Answer 1

An encoder is the input into a model being processed, whereas the decoder is the output of the model being generated.

Question 2

What is meant by neural attention?

Answer 2

Here we talk about neural networks that automatically weight input relevance when making predictions - they attend to the inputs. This is advantageous as we get performance gains.

Question 3

What is a ‘query vector’, q?

Answer 3

It is the query from the input sequence.

Question 4

What are ‘keys’?

Answer 4

Keys are the elements that are being matched or compared against the query vector, q.

Question 5

What are ‘values’? (Think keys, values, queries)

Answer 5

Values are the associated value to each key. The values represent context or meaning of the input elements.

Question 6

How do we normally transform affinity scores into probabilities?

Answer 6

We use the softmax, or the soft argmax, transformation.

Question 7

How do we compare a query, q and a key vector, h?

Answer 7

We use the dot product attention. The queries and keys need to have the same dimensionality.

Question 8

Keys and values needs to be the same vectors. True or false?

Answer 8

False! They can correspond to different linear projections of the same vectors, but they do not need to be the same vector.

Question 9

What is it called when we have multiple neural attention operations that are combined?

Answer 9

Multi-head attention.

Question 10

What is the process of dot-product self-attention?

Answer 10

First we compute the query vectors.
Then we compute the key vectors.
Then we compute the values vectors.
Then we compute the query-key affinity scores with dot products.
We convert these scores to probabilities through softmax.
Finally we output the weighted average of the values.

Question 11

How do we handle increasing complexity due to an increase in dimensionality of queries and keys?

Answer 11

We scale by the length of the query/key vectors. Remember: they have the same dimension.

Question 12

Attention(Q,K,V) = ?
In the above, Q = queries, K = keys, V = values

Answer 12

softmax( (QK^T / sqrt(d_k)) ) * V
where d_k is the dimension of the token vectors.

Question 13

What is masked self-attention?

Answer 13

It is similar to self-attention, except a part of the input is masked and then predicted at each step. For example if we are predicting the next word, we will want to mask that word that we are predicting.

Question 14

Why is masked self-attention useful?

Answer 14

It helps us understand meaning and context in sentences where there are missing words, and thus increases our prediction power. It achieves this by focusing on the important words and the relationship with other words in that sentence.

Question 15

Briefly explain how multi-head self-attention works.

Answer 15

In multi-head self-attention we split the input into smaller simultaneous sequences (on different splits) and perform self-attention. To get the final output we add or concatenate the outputs of each self-attention.

Question 16

If we have 6 attention heads (multi-head self-attention) and a token embedding size of 600, how might we project tokens to each attention head?

Answer 16

By sizes of 100 within each attention head.

Question 17

Why are transformers called transformers and what mechanism do they use to ‘transform’?

Answer 17

Because they utilize the transformer architecture. This archutecture uses self-attention to ‘transform’ input very effectively.

Question 18

Sequence-to-sequence (seq2seq) are based on self-attention. True or false?

Answer 18

True

Question 19

Transformers: Deep models with several layers scale quadratically with respect to sequence length. True or false?

Answer 19

True

Question 20

The encoder layers does not use self-attention and feed-forward transformations. True or false?

Answer 20

False. They use both.

Question 21

Decoder layers use masked self-attention for auto-regressive masking. True or false?

Answer 21

True

Question 22

Decoder layers uses cross-attention to consider context from the encoder states. True or false?

Answer 22

True

Question 23

What are encoder-only models good for? An example is the BERT.

Answer 23

They are really effective at solving classification and span prediction tasks ( question-answering, speech recognition etc.).

Question 24

What is transfer learning?

Answer 24

In machine learning, transfer learning is when a model learns from one task and then uses that learning to help with another related task. It’s like using what you already know to make learning new things faster and easier.

Question 25

What do we use encoder-decoder models for?

Answer 25

Generative and classification tasks. They are great for transfer learning.

Question 26

What do we use decoder-only models for?

Answer 26

This is the technology behind the large generative language models, such as the Chat GPT.

Question 27

What is ‘vector quantization’?

Answer 27

Vector quantization is a method of representing data by grouping similar vectors together and using a smaller set of representative vectors to approximate the original data.

Question 28

What do we use the Audio Spectrogram Transformer (AST) for?

Answer 28

Recognizing speech commands, language identification, speaker identification, speech emotion recognition, environmental audio classification.

Question 29

What is the ‘Mockingjay’ model?

Answer 29

It is a model that is inspired by BERT, and is pre-trained on a random subset of frames. The model tries to reconstruct an original frame.

Question 30

What is the DiscreteBERT?

Answer 30

It is a model that uses discrete units as input rather than continuous audio features. It avoids local smoothness!

Question 31

What is a ‘conformer’?

Answer 31

A conformer combines convolution layers to exploit local features with self-attention to model global context. It replaces the standard transformer blocks with ‘conformer blocks’.

Question 32

Transformers for audio input are very much different from transformers with text input. True or false?

Answer 32

False. They are similar in many ways.

Question 33

In what way is the original ‘wav2vec’ model trained?

Answer 33

Using contrastive predictive coding. And trained using self-supervised learning.

Question 34

What is the essense ‘Constrastive Predictive Coding’?

Answer 34

The CPC learns the current context representation from speech data. It does so, by predicting the future audio segments from past samples.

The loss implicitly models the mutual information between the context representations and the future encoder feature vectors.

For these reasons, it is good for speaker recognition.

Question 35

How does the ‘VQ-Wav2vec’ differ from the original ‘wav2vec’ model?

Answer 35

As the name infers, the VQ stands for Vector Quantization and uses this to transform acoustic features into discrete units.

Question 36

What special trick does the quantization step in the ‘VQ-Wav2vec’ make use of?

Answer 36

It uses the Gumbel-softmax trick for straight-through estimation. This gives a dense representation.

Question 37

Why is the HuBERT useful?

Answer 37

It is useful because it combines the benefits of both unsupervised and supervised learning, leveraging large-scale unlabeled data and labeled data to achieve state-of-the-art performance in speech recognition and related tasks.

Question 38

What is the OpenSMILE toolbox used for?

Answer 38

It is an open-source audio feature extractor. It stands for Speech & Music Interpretation by Large-space Extraction. It is mostly used in the area of paralinguistics.

Question 39

What is the ‘W2v-Conformer’ and why is it useful?

Answer 39

“W2v-Conformer” is useful because it combines the power of the Conformer architecture with the advantages of the word2vec pretraining approach, resulting in improved performance in automatic speech recognition and other natural language processing tasks.

Question 40

Why is the ‘BigSSL’ useful?

Answer 40

(Big Self-Supervised Learning) is useful because it leverages large-scale self-supervised pretraining on unlabeled data, enabling the model to learn meaningful representations and achieve better performance across various downstream tasks such as text classification, machine translation, and question-answering.

Question 41

What does the ‘W2v-BERT’ combine?

Answer 41

The contrastive loss of wav2vec with the masked language modelling of bert, without alternating objectives (like the HuBERT does).

Question 42

What does the SpeechT5 stand for? (the 5 T’s)

Answer 42

Text-To-Text Transfer Transformer.

Question 43

What is a ‘vocoder’?

Answer 43

Is is short for voice encoder and decoder and it is used to manipulate and synthesize speech and other audio.

Question 44

The SpeechT5 uses cross-modal mapping through vector quantization. True or false?

Answer 44

True

Question 45

What is the OpenAI Whisper model based on, or - what is the main task of the model?

Answer 45

Speech-to-text

Question 46

What type of model is the OpenAI Whisper model and what is the primary use?

Answer 46

It is a sequence-to-sequence transformer model trained on many different speech tasks. The primary uses are all based on speech-to-text.

Question 47

What is HuggingFace used for?

Answer 47

It is an open-source provider of NLP/ML technologies. It is also a library of transformer models to ude for example on top of pytorch.

Question 48

When we have multi-lingual speech data we might use VALL-E and VALL-E X. True or false?

Answer 48

True. The X stands for Cross-lingual.

Question 49

The VALL-E and VALL-E X uses EnCodec. What is this?

Answer 49

It is Convolutional auto-encoder. It creates audio from discrete tokens using this method.

Question 50

What kind of transformers do VALL-E combine?

Answer 50

The main model combines autoregressive/non-autoregressive Transformers.