Natural language processing: An unexpected journey

Question 1

What is natural language processing?

Answer 1

Is a field of artificial intelligence that allows machines to read, understand and derive meaning from human languages.

(Terms ‘natural language processing’, ‘computational linguistics’ and ‘human language technologies’ may be thought of as essentially synonymous.)

Question 2

Why is natural language processing tricky?

Answer 2

Text data is fundamentally discrete. But new words can always be created.

• Zipf’s law and Herdan’s / Heaps’ law
• Language is ambiguous, compositional, recursive, unveils hidden structure

Question 3

Zipf’s law and Herdan’s law

Answer 3

• Few words are very frequent, and there is a long tail of rare words (Zipf’s law)
• Out-of-vocabulary words are always being discovered (Herdan’s /Heaps’ law). There is no way to discover the full vocabulary from which the text documents are drawn

(draw the graphs)

Question 4

Ambiguity, composition, recursion and hidden structure (units)

Answer 4

• Language is ambiguous: units can have different meanings (e.g. “fans”)
• Language is compositional: meaning of a unit defined as a function of the meaning of its components (think about the phrase structure levels)
• Language is recursive: phrases can be repeatedly combined (think about this video https://youtu.be/MPWuI9whbEY)
• Language unveils hidden structure: local changes in a sentence might have global effects (e.g. The trophy doesn’t fit into the brown suitcase because it is too {small, large}.)

(think of some examples)

Question 5

Generative Linguistics

Answer 5

Is a linguistic theory that argues for the existence of a language faculty in all human beings, which encodes a set of abstractions specially designed to facilitate the understanding and production of language.

Question 6

NLP: Empiricism vs Rationalism

Answer 6

• Empiricism: view that there is no such thing as innate knowledge, and that knowledge is instead derived from experience (machine learning)
• Rationalism: a significant part of the knowledge in the human mind is not derived by the senses but is fixed in advance, presumably by genetic inheritance (linguistic knowledge)

Question 7

NLP: optimization problem, search module and learning module

Answer 7

Many natural language processing problems can be written mathematically in the form of optimization:

write the formula…

• The search module is responsible for finding the candidate output ŷ with the highest score relative to the input x.
• The learning module is responsible for finding the model parameters θ that maximizes the predictive performance