07 - End-2-End Speech Recognition

Question 1

Let us start with a reminder. What is an end-2-end system or model?

Answer 1

A sytem or model that takes in an input and directly models an output without relying on intermediate stages or processes.

Question 2

HMM-based problems are normally split into 3 separate models, which are?

Answer 2

The Acoustic Model (AM), the Language Model (LM) and the Pronunciation Model (PM).

Question 3

HMM is quick and efficient and does not require expert knowledge. True or false?

Answer 3

False. HMM’s require expert knowledge and is very time consuming because of the 3 models that underly in the structure.

Question 4

In the acoustic model (that is the sound signal input) what are the main issues (there are 3)?

Answer 4

1. The input is of variable length.
2. The input is often much larger than the output.
3. We do not know how the input audio features align with the output characters.

Question 5

What are the two main architectures used to solve the ‘alignment problem’ in end-2-end models?

Answer 5

The CTC (Connectionist Temporal Classification) and the ‘seq2seq-attention’

Question 6

The CTC was proposed in 2006 as a model that can train an acoustic model without requiring segmentation and alignment. True or false?

Answer 6

True.

Question 7

An original approach in speech recognition outputted phonemes and NOT words. Is it still an end-2-end model when the output is phonemes?

Answer 7

NO. It is not the complete process. You would have to transfer the phonemes into words afterwards thus relying on an intermediate process.

Question 8

The CTC does not allow for identifying consecutive character repetitions. For example in the word ‘hello’ (2 l’s). What special token does CTC introduce to combat this?

Answer 8

The blank (ϵ) token.

Question 9

The CTC mapping makes use of the blank token (ϵ) to comprehend what?

Answer 9

An instance where a word has two consecutive letters. This is because input and output can have different lengths.

Question 10

The probability of an alignment in the CTC is what?

Answer 10

The dot-product of the probabilities at each time step.

Question 11

CTC can perform inference, and uses two search mechanisms. These are?

Answer 11

The greedy search and the beam search

Question 12

What is the ‘Greedy Search’?

Answer 12

This takes the most likely output at each time-step. It will give us the alignment with the highest probability.

Question 13

What is the ‘Beam Search’?

Answer 13

It computes a new set of hypotheses at each input step with all possible combinations, but keeps only the top candidates.

Question 14

All network outputs of the CTC are conditionally dependent. True or false?

Answer 14

False. They are conditionally independent.

Question 15

Is CTC (used alone) a real end-2-end approach?

Answer 15

No. Because in order to obtain good performance it relies on the use of external language models.

Question 16

The ‘seq-2-seq-attention’ architecture is made of 3 blocks. These are?

Answer 16

An encoder
An attention mechanism
A decoder

Question 17

In ‘seq-2-seq-attention’ what is the role of the encoder?

Answer 17

The encoder is analogous to an Acoustic Model (AM) and transforms speech features into a higher level representation.

Question 18

In ‘seq-2-seq-attention’ what is the role of the attention mechanism?

Answer 18

This is analogous to an Alignment Model that chooses encoded frames that are relevant to produce output.

Question 19

In ‘seq-2-seq-attention’ what is the role of the decoder?

Answer 19

It is analogous to a Language Model (LM) that predicts each token as a function of the previous predictions and outputs it (after having applied softmax).

Question 20

Often dimensionality can be a problem in the encoder part of ‘seq-2-seq-attention’. It is common to use what NN in the first layers?

Answer 20

A Convolution Neural Network. After that you might apply BLSTM’s (bi-directional LSTM) to reduce the encoded features.

Question 21

The first example of a ‘seq-2-seq-attention’ model was the Listen, Attend and Spell. The output here were characters and not words. Was is still an end-2-end model if the desired output was characters?

Answer 21

Yes, but if the desired output was words, then no.

Question 22

In ‘seq-2-seq-attention’ a problem is that it is easily affected by noise. True or false?

Answer 22

True

Question 23

In ‘seq-2-seq-attention’ it is better to train with longer input sequences first and then shorter input sequences. True or false?

Answer 23

False. You want to train with the short input sequences first.

Question 24

An encoder-decoder approach can be ‘streamed’. True or false?

Answer 24

False. The encoder needs to complete the input, before the decoder can start working.

Question 25

In CTC, seq2seq and the hybrid CTC/Attention, the ‘beam search’ or the ‘greedy search’ is more common?

Answer 25

The Beam Search

Question 26

The ‘Hybrid CTC/attention decoding’ is more robust due to?

Answer 26

It is better with appropriate alignments in noisy environments. It uses Joint decoding during recognition.

Question 27

What is the ESPnet Toolkit used for?

Answer 27

It is dedicated to end-2-end speech processing!

Question 28

What can the RNN-T do that the CTC and other cannot?

Answer 28

It can stream due to its use of a joint network.

Question 29

Summary question: End-2-end is the current trend in the ASR field. True or false?

Answer 29

True

Question 30

End-2-end systems are better when there are limited resources compared to the ‘conventional’ HMM-DNN systems. True or false?

Answer 30

False