Lecture 9 - Multimodality in Speech

Question 1

What is the SNR range in speech intellgibility?

Answer 1

6dB to -5dB

Question 2

When observing the TCD-TIMIT set on lip reading, what features faired best and worst?

Answer 2

1. Best; thinner lips, same colour around mouth
2. Worst; facial hair, thicker lips

Question 3

The ‘LRS2’ dataset is from the BBC; why is it more challenging?

Answer 3

1. Head pose
2. Illumination changes
3. Low image resolution

Question 4

Explain why an approach of waiting for gaps in a conversation to indicate turns is not valid.

Answer 4

1. Gaps in real-life are short and overlap occurs in turn-taking.
2. The modal response is 200 ms and shown to take 600 ms to find term; we plan our term.

Question 5

Outline 4 cues which are used by humans to regulate turns in a conversation, and indicate whether they are used differently for a hold or yield.

Answer 5

1. Verbal;
   - yield; finishes sentence.
   - hold; filled pause
2. Prosody/Pitch;
   - yield; rising or falling pitch
   - hold; flat pitch
3. Breathing
   - yield; breathe out
   - hold; breathe in
4. Gaze
   - yield; look at addressee
   - hold; looking away

Question 6

What is back-channel?

Answer 6

Saying phrases such as ‘okay’ and ‘okay you go’