NLP tasks and evaluation

Question 1

What is NLP?

Answer 1

Making the machine understand human language

Question 2

What is text classification? Give examples

Answer 2

Assigning some label to text
- Sentiment classification (positive or negative text)
- SNLI (labeled by experts in contrary to IMDB dataset) = sentence-pair classification

Question 3

What is SNLI?

Answer 3

standard (Stanford) natural language inference

We have two sentences and we have to assign label to them: entailment, contradiction or neutral. SNLI is a dataset from Stanford, about 570k sentence pairs manually labeled for balanced classification

Question 4

How to measure task subjectivity and annotation quality?

Answer 4

Using inter-annotator agreement = take chance agreement into account. There are some metrics like Cohen’s Kappa or Scott’s Pi which are very pessimistic measures

Question 5

What is the gold standard? How is it produced?

Answer 5

This is the true label, usually labeled by some experts. It is super costly. It is calculated by majority voting between multiple annotators

Question 6

What is NER?

Answer 6

Named Entity Recognition: find entities withing text of predefined types (entities). When having multi-token entities of the same type, we use BEGINNING tag to show the start of a new entity

Question 7

What is SuperGLUE?

Answer 7

It is a popular benchmark collection of various tasks in English (9 hard NLP tasks) used for any method/model capable of being applied to a broad range of language understanding tasks.

Question 8

What is RTE?

Answer 8

Recognizing Textual Entailment (RTE) is a binary classification - whether or not the text entails the hypothesis

Question 9

What is Coreference resolution task

Answer 9

Winograd Schema Challenge: Examples with a pronoun and a lot of noun phrases from the sentence, and the task is to determine the correct referrent of the pronoun (who is he?). Requires everyday knowledge to solve

Question 10

What is BoolQ task?

Answer 10

We have a yes/no question and a short passage and we have to determine the answer to the question (yes/no). Requires difficult entailment-like inference, complex, can be ambigous

Question 11

Explain the MultiRC task

Answer 11

Multi-Sentence Reading Comprehension:
Each example consists of context paragraph, question about the paragraph and a list of possible answers (true/false). Task is to label each answers as TRUE or FALSE (there can be multiple TRUE ones)

Question 12

Explain the SQuAD 2.0 task

Answer 12

We have a text and a list of unanswerable questions with plausible but incorrect answers in the text. The model should tell you that it is not possible to know the answer from this text.

Question 13

What are some text generation tasks in NLP?

Answer 13

Machine translation (from language 1 to language 2). Difficult because people can disagree if the translation is accurate or not

Document summarization: given some text, make a short summary

PersonaChat: There are multiple personas with some descriptions and the model should continue the converstaion between these two personas

Question 14

What is classification as generation?

Answer 14

Nowadays, you can ask the generative model to label some text (classification). “Is the sentiment positive or negative”.

Question 15

What is k-fold cross-validation?

Answer 15

We have our dataset of N row and we partition the data into K chunks. K-1 of them are used for training and 1 is used for testing (or validation). This is repeated K times with each time a different partition is used for testing, and the final metric is the average of these K trained models. Before taking new partitioning, the model is reset to avoid data leakage (seeing test during the training)

Question 16

What is the confusion matrix?

Answer 16

It is a 2x2 matrix of TP, TN, FP, FN
- TP: Model correctly classified positive
- TN: Model correctly classified negative
- FP: Model incorrectly classified positive
- FN: Model incorrectly classified negative

Question 17

Explain the metric Accuracy. Why is it not enough to only use Accuracy?

Answer 17

It is how many examples the model correctly classified (TP + TN) divided by the number of all examples (TP + TN + FP + FN).

It is bad with huge class imbalances. If there are 1000 true cases and 50 false, if the model only predicts true, we get a high accuracy but the model is deterministic.

Question 18

Explain Precision and Recall metrics

Answer 18

Precision is how many of the predicted positives (TP + FP) are correctly predicted (TP) - quality
Recall is how many of the actual positives (TP + FN) are correctly predicted (TP) - quantity

Recall can be very high if we label all examples as positives (all are correctly labeled), but the precision will drop.
If we only predict super obvious ones (low number of them), the precision is high but recall is low.

That’s why we need some combined metric (F1 score)

Question 19

What is F1 score?

Answer 19

It is a combination of recall and precision:

2 * (P * R) / (P + R).

It reduces the inconsistancies of precision and recall.

Question 20

How would you calculate Precision, Recall and F1 score?

Answer 20

Question 21

What changes with multi-class classification confusion matrix?

Answer 21

Basically we can convert it to 2x2 by considering everything that is classified != true label as negative.

Question 23

How to do evaluation of MT systems?

Answer 23

We can use metric like BLEU (Bilingual Evaluation Understudy) for Machine Translation (MT) tasks.

It computes some precision-based metric on n-gram overlap between true and inferred text. In particular, BLEU is the ration of the number of overlapping n-grams to the noral number of n-grams in the hypothesis

Con: translation is ambiguous and doesn’t take recall into account

Question 24

Explain ROUGE evaluation metric

Answer 24

Recall-Oriented Understudy for Gisting Evaluation:
Similar to BLEU but it is recall-based instead of precision-based. It counts n-grams as well