Introduction - Week 1

Question 1

What makes an application a language processing application

Answer 1

It requires the use of knowledge about human language

Question 2

Is Unix wc an example of a language processing application?

Answer 2

Yes, when it counts words
No, when it counts lines or bytes. Lines and bytes are computer artefacts, not linguistic entities

Question 3

Is google search an NLP application?

Answer 3

Yes, it uses knowledge about human languages

Question 4

Why is NLP hard?

Answer 4

Text is only structured for the human user, often almost fully unstructured for the machine, sometimes ‘semi-structured’ like html

Question 5

semi-structured text

Answer 5

Text that is partially structured for the machine like HTML

Question 6

Natural Language Processing (NLP)

Answer 6

Necessary steps for “understanding” a piece of data represented by a language

Question 7

NLP tasks (umbrella terms)

Answer 7

• Text mining
• Text analytics
• Computational Linguistics
• (human) language technology

Question 8

NLP Tasks and Applications

Answer 8

Information Retrieval
- Searching for relevant documents

Document classification
- Sorting documents into categories

Question answering
- Short answer for a question

Text summarisation
- Summarise a set of documents

Sentiment analysis
- Product reviews, Twitter, Hate crime detection

Machine translation
- One of the first motivations for NLP

Natural language generation
- For data to text

Authoring and marking tools
- Check spelling, grammar, style
- Automated marking of essays

Conversational Agents
- Dialogues, voice recognition, Text to speech, speech to Text

etc… (many many others)

Question 9

NLP main problems

Answer 9

Variability
Ambiguity

Question 10

Variability

Answer 10

Numerous ways to say the same thing

Question 11

Ambiguity

Answer 11

Words and sentences are often ambiguous, and can have multiple meanings

Question 12

Word-level ambiguity

Answer 12

Apple (company) or Apple (fruit)

Question 13

Sentence-level ambiguity

Answer 13

I made her duck (this has at least 5 meanings)

Question 14

Lexical Ambiguity

Answer 14

A word with multiple POS tags, e.g. Duck can be verb or noun

Question 15

Lexical-semantic ambiguity

Answer 15

A word with different senses
e.g. bank can be financial institution or part of countryside (river bank)

Question 16

Syntactic Ambiguity

Answer 16

Ambiguity combing from possible word groupings

Question 17

Parts of Speech (POS)

Answer 17

nouns, verbs, adjectives, adverbs, pronouns, prepositions, auxiliaries, determiners

Question 18

Open class words

Answer 18

nouns, verbs, adjectives, adverbs

Question 19

Closed class words

Answer 19

pronouns, prepositions, auxiliaries, determiners

Question 20

Attachment Ambiguity

Answer 20

I saw the girl with the telescope
(did he see the girl through a telescope, or see the girl using the telescope?)

Question 21

Coordination ambiguity

Answer 21

Old men and women (are the women also old?)

Mother and baby in pram (is the mother in the pram?)

Question 22

Local Ambiguity

Answer 22

Police help dog bite victim (Are the police also biting the victim?)

Question 23

Corpus

Answer 23

a (large) collection of linguistic data
- May consist of written texts, spoken discourse, samples of spoken or written language

Question 24

unannotated corpus

Answer 24

raw text/speech

Question 25

annotated (labelled) corpus

Answer 25

raw text/speech enhanced with linguistic information

A repository of explicit linguistic information (added manually or automatically)
e.g. specifying that “loves” in “Mary loves John” is 3rd person singular present tense form of a verb

Question 26

Corpus annotation types

Answer 26

Grammatical (e.g. POS tags, noun / verb phrases)

Semantic (e.g. person, drug)

Pragmatics - language in use (e.g. conversation)

Combined

Question 27

Why do we need annotated corpora?

Answer 27

For training and evaluation

Training:
- Train linguists, language learners, etc…
- Researchers (e.g. in language development)
- NLP development: use ML/statistics to learn patterns from an annotated corpus

NLP evaluation:
- Compare NLP results (automated “annotations”) with a manually coded “gold standard”

Question 28

Do people agree on annotations?

Answer 28

No, sometimes very subjective and inconsistent, very difficult to get a gold standard corpus

Some simple tasks are relatively consistent, e.g. what’s the name of the lecture?
Or inconsistent, e.g. sentiment analysis

Question 29

Annotation agreement

Answer 29

Kappa - measures the agreement between two classifiers, who classify N items into C mutually exclusive categories

k = (Pr(a) - Pr(e)) / (1-Pr(e))

Pr(a) = relative observed agreement among annotators

Pr(e) = hypothetical probability of chance agreement - typically uses the observed data to calculate the probabilities of each observer randomly saying each category

If in complete agreement then k=1, if completely disagree, other than what would be expected by chance (as defined by Pr(e)) then k=0

Question 30

Precision

Answer 30

Fraction of retrieved documents that are relevant

relevant items retrieved / retrieved items

P(relevant|retrieved)

Question 31

Recall

Answer 31

Fraction of relevant documents that are retrieved

relevant items retrieved / relevant items

P(retrieved | relevant)

Question 32

F-measure

Answer 32

Weighted harmonic mean between Precision and Recall, trades-off the two

F = 2PR/(P + R)

Question 33

Cross-validation

Answer 33

Break up data into n folds
- Equal positive and negative in each fold?

For each fold:
- Choose the fold as a temporary test set
- Train on the other n-1 folds, compute performance on the test fold

Report average performance of the n runs

Sensible value for n might be 10